So I’m no expert, but I have been a hobbyist C and Rust dev for a while now, and I’ve installed tons of programs from GitHub and whatnot that required manual compilation or other hoops to jump through, but I am constantly befuddled installing python apps. They seem to always need a very specific (often outdated) version of python, require a bunch of venv nonsense, googling gives tons of outdated info that no longer works, and generally seem incredibly not portable. As someone who doesn’t work in python, it seems more obtuse than any other language’s ecosystem. Why is it like this?

  • nickwitha_k (he/him)
    link
    fedilink
    7015 days ago

    Python’s packaging is not great. Pip and venvs help but, it’s lightyears behind anything you’re used to. My go-to is using a venv for everything.

  • @[email protected]
    link
    fedilink
    5915 days ago

    It’s something of a “14 competing standards” situation, but uv seems to be the nerd favourite these days.

    • @[email protected]
      link
      fedilink
      English
      3015 days ago

      I still do the python3 -m venv venv && source venv/bin/activate

      How can uv help me be a better person?

      • NostraDavid
        link
        fedilink
        714 days ago
        1. let pyproject.toml track the dependencies and dev-dependencies you actually care about
        • dependencies are what you need to run your application
        • dev-dependencies are not necessary to run your app, but to develop it (formatting, linting, utilities, etc)
        1. it can track exactly what’s needed ot run the application via the uv.lock file that contains each and every lib that’s needed.
        2. uv will install the needed Python version for you, completely separate from what your system is running.
        3. uv sync and uv run <application> is pretty much all you need to get going
        4. it’s blazingly fast in everything
      • @[email protected]
        link
        fedilink
        315 days ago

        If you’re happy with your solution, that’s great!

        uv combines a bunch of tools into one simple, incredibly fast interface, and keeps a lock file up to date with what’s installed in the project right now. Makes docker and collaboration easier. Its main benefit for me is that it minimizes context switching/cognitive load

        Ultimately, I encourage you to use what makes sense to you tho :)

    • @[email protected]
      link
      fedilink
      215 days ago

      This! Haven’t used that one personally, but seeing how good ruff is I bet it’s darn amazing, next best thing that I used has been PDM and Poetry, because Python’s first party tooling has always been lackluster, no cohesive way to define a project and actually work it until relatively recently

      • NostraDavid
        link
        fedilink
        414 days ago

        I bet it’s darn amazing,

        It is. In this older article (by Anna-Lena Popkes) uv is still not in the middle, but I would claim it’s the new King of Project Management, when it comes to Python.

        uv init --name <some name> --package --app and you’re off to the races.

        Are you cloning a repo that’s uv-enabled? Just uv sync and you’re done!

        Heck, you can now add dependencies to a script and just uv run --script script.py (IIRC) and you don’t need to install anything - uv will take care of it all, including a needed Python version.

        Only downside is that it’s not 1.0 yet, so the API can change at any update. That is the last hurdle for me.

      • @[email protected]
        link
        fedilink
        4
        edit-2
        15 days ago

        I moved all our projects (and devs) from poetry to uv. Reasons were poetry’s non standard pyproject.toml syntax and speed, plus some weird quirks, e. g. if poetry asks for input and is not run with the verbose flag, devs often don’t notice and believe it is stuck (even though it’s in the default project README).

        Personally, I update uv on my local machine as soon as a new release is available so I can track any breaking changes. Couple of months in, I can say there were some hiccups in the beginning, but currently, it’s smooth sailing, and the speed gain really affects productivity as well, mostly due to being able to not break away from a mental “flow” state while staring at updates, becoming suspicious something might be wrong. Don’t get me wrong, apart from the custom syntax (poetry partially predates the pyproject PEP), poetry worked great for us for years, but uv feels nicer.

        Recently, “uv build” was introduced, which simplified things. I wish there was an command to update the lock file while also updating the dependency specs in the project file. I ran some command today and by accident discovered that custom dependency groups (apart from e. g. “dev”) have made it to uv, too.

        “uv pip” does some things differently, in particular when resolving packages (it’s possible to switch to pip’s behavior now), but I do agree with the decisions, in particular the changes to prevent “dependency confusion” attacks.

        As for the original question: Python really has a bit of a history of project management and build tools, I do feel however that the community and maintainers are finally getting somewhere.

        cargo is a bit of an “unfair” comparison since its development happened much more aligned with Rust and its whole ecosystem and not as an afterthought by third party developers, but I agree: cargo is definitely a great benchmark how project and dependency management plus building should look like, along with rustup, it really makes the developer experience quite pleasant.

        The need for virtual environments exists so that different projects can use different versions of dependencies and those dependencies can be installed in a project specific location vs a global, system location. Since Python is interpreted, these dependencies need to stick around for the lifetime of the program so they can be imported at runtime. poetry managed those in a separate folder in e. g. the user’s cache directory, whereas uv for example stores the virtual environment in the project folder, which I strongly prefer.

        cargo will download the matching dependencies (along with doing some caching) and link the correct version to the project, so a conceptual virtual environment doesn’t need to exist for Rust. By default, rust links everything apart from the C runtime statically, so the dependencies are no longer neesed after the build - except you probably want to rebuild the project later, so there is some caching.

        Finally, I’d also recommend to go and try setting up a project using astral’s uv. It handles sane pyproject.toml files, will create/initialize new projects from a template, manages virtual environments and has CLI to build e. g. wheels or source distribution (you will need to specify which build backend to use. I use hatchling), but thats just a decision you make and express as one line in the project file. Note: hatchling is the build backend, hatch is pypa’s project management, pretty much an alternative to poetry or uv.

        uv will also install complete Python distributions (e. g. Python 3.12) if you need a different interpreter version for compatibility reasons

        If you use workspaces in cargo, uv also does those.

        uv init, uv add, uv lock --upgrade, uv sync, uv build and how uv handles tools you might want to install and run should really go a long way and probably provide an experience somewhat similar to cargo.

        • @[email protected]
          link
          fedilink
          115 days ago

          I think you responded to the wrong comment, I didn’t question the need for uv or other tools like that

          • @[email protected]
            link
            fedilink
            314 days ago

            I did that on purpose, i. e. I wanted to confirm your thoughts about uv, drifted off into a general rant, remembered OP’s original question and later realized it would have been better framed as a top level comment. In my defense, I was in an altered state of mind at the time.

  • @[email protected]
    link
    fedilink
    English
    5214 days ago

    Python is the only programming language that has forced me to question what the difference is between an egg and a wheel.

  • @[email protected]
    link
    fedilink
    5015 days ago

    You re not stupid, python’s packaging & versionning is PITA. as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem

    • @[email protected]
      link
      fedilink
      1615 days ago

      as long as you write it for yourself, you re good. As soon as you want to share it, you have a problem

      A perfect summary of the history of computer code!

  • JackbyDev
    link
    fedilink
    English
    35
    edit-2
    14 days ago

    No, it’s not just you, Python’s tooling is a mess. It’s not necessarily anyone’s fault, but there are a ton of options and a lot of very similarly named things that accomplish different (but sometimes similar) tasks. (pyenv, venv, and virtualenv come to mind.) As someone who considers themselves between beginner and intermediate proficiency in Python, this is my biggest hurdle right now.

    • NostraDavid
      link
      fedilink
      1514 days ago

      Python’s tooling is a mess.

      Not only that. It’s a historic mess. Over the years, growing a better and better toolset left a lot of projects in a very messy state. So many answers on Stack Overflow that mention easy_install - I still don’t know what it is, but I guess it was some kind of proto uv.

      • JackbyDev
        link
        fedilink
        English
        614 days ago

        Every time I’m doing anything with Python I ask myself if Java’s tooling is this complicated or I’m just used to it by now. I think a big part of the weirdness is that a lot of Python tooling is tied to the Python installation whereas in Java things like Maven and Gradle are separate. In addition, I think dependencies you install get tied to that Python installation, while in Java they just are in a cache for Maven/Gradle. And in the horrible scenario where you need to use different versions of Maven/Gradle (one place I was at specifically needed Maven 3.0.3 for one project and a different for a different, don’t ask, it’s dumb and their own fault for setting it up that way) at least they still have one common cache for everything.

        I guess it also helps that with Java you (often) don’t need platform specific jar files. But Python is often used as an easy and dynamic scripting interface over more performant, native code. So you don’t really run into things like “this artifact doesn’t have a 64 bit arm version for python 2” often with Java. But that’s not a fault of Python’s tooling, it’s just the reality of how it’s used.

  • @[email protected]
    link
    fedilink
    3015 days ago

    Yes it’s terrible. The only hope on the horizon is uv. It’s significantly better than all the other tooling (Poetry, pip, pipenv, etc.) so I think it has a good chance of reducing the options to just Pip or uv at least.

    But I fully expect the Python Devs to ignore it, and maybe even make life deliberately difficult for it like they did for static analysers. They have some strange priorities sometimes.

    • @[email protected]
      link
      fedilink
      8
      edit-2
      15 days ago

      I like the idea of uv, but I hate the name. Libuv is already a very popular C library, and used in everything from NodeJS to Julia to Python (through the popular uvloop module). Every time I see someone mention uv I get confused and think they’re talking about uvloop until I remember the Astral project, and then reconfirm to myself how much I disapprove of their name choice.

      • @[email protected]
        link
        fedilink
        214 days ago

        I don’t think libuv is really that popular, nor is it that confusing.

        But I do agree it’s not a very good name. “Rye” is a much better name. Probably too late anyway.

    • @[email protected]
      link
      fedilink
      114 days ago

      uv is good but it needs a little more time in the oven.

      For the moment I would definitely recommend poetry if you are not a library developer. Poetry’s biggest sin is it’s atrocious performance but it has most of the features you need to work with Python apps today.

      • @[email protected]
        link
        fedilink
        314 days ago

        Why do you say it needs more time in the oven? I’ve had zero issues with it as a drop-in replacement for Pip in a large commercial project, which is an extremely impressive achievement. (And it was 10x faster.)

        I tried Poetry once and it failed to resolve dependencies on the first thing I tried it on. If anything Poetry needs more time in the oven. It also wasn’t 10x faster.

  • @[email protected]
    link
    fedilink
    English
    27
    edit-2
    15 days ago

    Python developer here. Venv is good, venv is life. Every single project I create starts with

    python3 -m venv venv

    source venv/bin/activate

    pip3 install {everything I need}

    pip3 freeze > requirements.txt

    Now write code!

    Don’t forget to update your requirements.txt using pip3 freeze again anytime you add a new library with pip.

    If you installed a lot of packages before starting to develop with virtual environments, some libraries will be in your OS python install and won’t be reflected in pip freeze and won’t get into your venv. This is the root of all evil. First of all, don’t do that. Second, you can force libraries to install into your venv despite them also being in your system by installing like so:

    pip3 install --ignore-installed mypackage

    If you don’t change between Linux and windows most libraries will just work between systems, but if you have problems on another system, just recreate the whole venv structure

    rm -rf venv (…make a new venv, activate it) pip3 install -r requirements.txt

    Once you get the hang of this you can make Python behave without a lot of hassle.

    This is a case where a strength can also be a weakness.

    • NostraDavid
      link
      fedilink
      2014 days ago

      pip3 freeze > requirements.txt

      I hate this. Because now I have a list of your dependencies, but also the dependencies of the dependencies, and I now have regular dependencies and dev-dependencies mixed up. If I’m new to Python I would have NO idea which libraries would be the important ones because it’s a jumbled mess.

      I’ve come to love uv (coming from poetry, coming from pip with a requirements/base.txt and requirements/dev.txt - gotta keep regular dependencies and dev-dependencies separate).

      uv sync

      uv run <application>

      That’s it. I don’t even need to install a compatible Python version, as uv takes care of that for me. It’ll automatically create a local .venv/, and it’s blazingly fast.

      • @[email protected]
        link
        fedilink
        English
        214 days ago

        I’ve never really spent much time with uv, I’ll give it a try. It seems like it takes a few steps out of the process and some guesswork too.

    • JackbyDev
      link
      fedilink
      English
      7
      edit-2
      14 days ago

      Okay, now give me those steps but what to do if I clone an already existing repo please

      • @[email protected]
        link
        fedilink
        414 days ago

        The git repo should ignore the venv folder, so when you clone you then create a new one and activate it with those steps.

        Then when you are installing requirements with pip, the repo you cloned will likely have a requirements.txt file in it, so you ‘pip install -r requirements.txt’

    • @[email protected]
      link
      fedilink
      613 days ago

      You have been in lala land for too long. That list of things to do is insane. Venv is possibly one of the worst solutions around, but many Python devs are incapable of seeing how bad it is. Just for comparison, so you can understand, in Ruby literally everything you did is covered by one command bundle. On every system.

    • @[email protected]
      link
      fedilink
      614 days ago

      OP sounds like a victim of Python 3, finding various Python 2 projects on the internet, a venv isn’t going to help

  • Ephera
    link
    fedilink
    2315 days ago

    Python never had much of a central design team. People mostly just scratched their own itch, so you get lots of different tools that do only a small part each, and aren’t necessarily compatible.

  • lime!
    link
    fedilink
    English
    2315 days ago

    everyone focuses on the tooling, not many are focusing on the reason: python is extremely dynamic. like, magic dynamic you can modify a module halfway through an import, you can replace class attributes and automatically propagate to instances, you can decompile the bytecode while it’s running.

    combine this with the fact that it’s installed by default and used basically everywhere and you get an environment that needs to be carefully managed for the sake of the system.

    js has this packaging system down pat, but it has the advantage that it got mainstream in a sandboxed isolated environment before it started leaking out into the system. python was in there from the beginning, and every change breaks someone’s workflow.

    the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.

    • @[email protected]
      link
      fedilink
      113 days ago

      and yet that all works fine in Ruby, which came out around the same time as Python and yet has had Bundler for 15 years now.

      Python - 15+ package managers and build tools Ruby - 1

      the closest language to look at for packaging is probably lua, which has similar issues. however since lua is usually not a standalone application platform it’s not a big deal there.

      no the closest language is literally Ruby, it’s almost the exact same language, except the tooling isn’t insane and it came out only a few years after python.

      • lime!
        link
        fedilink
        English
        213 days ago

        good point, ruby is a good comparison. although, ruby is very different under the hood. it’s magically dynamic in a completely different way, and it also never really got the penetration on the system level that python did.

        none of this is to take away from the fact that python packaging is bad. i know how to work it because i’ve been programming in python for 14 years, but trying to teach people makes the problem obvious. and yet.

  • @[email protected]
    link
    fedilink
    20
    edit-2
    15 days ago

    The reason you do stuff in a venv is to isolate that environment from other python projects on your system, so one Python project doesn’t break another. I use Docker for similar reasons for a lot of non-Python projects.

    A lot of Python projects involve specific versions of libraries, because things break. I’ve had similar issues with non-Python projects. I’m not sure I’d say Python is particularly worse about it.

    There are tools in place that can make the sharing of Python projects incredibly easy and portable and consistent, but I only ever see the best maintained projects using them unfortunately.

  • @[email protected]
    link
    fedilink
    English
    1715 days ago

    I agree. Python is my language of choice 80% or so of the time.

    But my god, it does packaging badly! Especially if it’s dependent on linking to compiled code!

    Why it is like that, I couldn’t tell. The language is older than git, so that might be part of it.

    However, you’re installing python libraries from github? I very very rarely have to do that. In what context do you have to do that regularly?

  • @[email protected]
    link
    fedilink
    1715 days ago

    Python is hacky, because it hacks. There’s a bunch of ways you can do anything. You can run it on numerous platforms, or even on web assembly. It’s not maintained centrally. Each “app” you find is just somebodies hack project they’re sharing with you for fun.

        • magic_lobster_party
          link
          fedilink
          314 days ago

          Nothing comes close to Perl’s abuse of global variables. Oh you called this function? Take a guess which global variables it will use.

        • Billegh
          link
          fedilink
          115 days ago

          Yes. Its line noise was of a much higher quality. 😉

      • @[email protected]
        link
        fedilink
        114 days ago

        On that note, I’m hesitant between writing my scripts in perl or python right now. Bash prevent sharing with Windows peoples… I just want to provide easy wrappers tools that are usually aroud 10 lines of shell, but testers ain’t on linux so they cannot use them.

        I don’t know perl, but each time I interract with pyton’s projects I have a different venv/poetry/… to setup. Forget adout it the next time and nothing is kept easy to reuse.

        • Billegh
          link
          fedilink
          214 days ago

          Perl isn’t really any better. There aren’t easy tools that do the same thing as venv. They exist, but they are not easy. Plus there are a much larger amount of cpan modules that have c in them than python.

      • @[email protected]
        link
        fedilink
        515 days ago

        I’ve been full time writing python professionally since 2015. You get used to it. It starts to just make sense and feel normal. Then when you move to a different language environment you wonder why their tooling doesn’t use a virtualenv.

        • Lettuce eat lettuce
          link
          fedilink
          114 days ago

          I’m starting to get the hang of it. I was using Debian, so I had to figure out the basics of venv because many of the frameworks I was trying to learn require newer versions of Python than what comes with Debian.

          vscodium works really easily inside it though, so it wasn’t too bad, but I still feel like I’m treading water a little bit.

  • @[email protected]
    link
    fedilink
    English
    1315 days ago

    With all the hype surrounding Python it’s easy to forget that it’s a really old language. And, in my opinion, the leadership is a bit of a mess so there hasn’t been any concerted effort on standardizing tooling.

    Some unsolicited advice from somebody who is used more refined build environments but is doing a lot of Python these days:

    The whole venv thing isn’t too bad once you get the hang of it. But be prepared for people to tell you that you’re using the wrong venv for reasons you’ll never quit understand or likely need to care about. Just use the bundled “python -m venv venv” and you’ll be fine despite other “better” alternatives. It’s bundled so it’s always available to you. And feel free to just drop/recreate your venv whenever you like or need. They’re ephemeral and pretty large once you’ve installed a lot of things.

    Use “pipx” to install python applications you want to use as programs rather than libraries. It creates and manages venvs for them so you don’t get library conflicts. Something like “pip-tools” for example (pipx install pip-tools).

    Use “pyenv” to manage installed python versions - it’s a bit like “sdkman” for the JVM ecosystem and makes it easy to deal with the “specific versions of python” stuff.

    For dependencies for an app - I just create a requirements.txt and “pip install -r requirements.txt” for the most part… Though I should use one of the 80 better ways to do it because they can help with updating versions automatically. Those tools mostly also just spit out a requirements.txt in the end so it’s pretty easy to migrate to them. pip-tools is what my team is moving towards and it seems a reasonable option. YMMV.

    • @[email protected]
      link
      fedilink
      English
      2
      edit-2
      14 days ago

      This.

      venv
      pip-tools

      Specify your primary dependencies in pyproject.toml and use pip-compile to keep stuff locked in requirements.txt to exact versions (or even hashes).
      Though after working with cargo a bit, I would love to have all of this in a first-class program, hope uv can get there.