wip: add a create_distribution entry point#1073
Conversation
this creates basically a new julia "distribution" with a custom set of packages in it. Right now: - packages get added to the sysimage - packages are treated as stdlib so they can be loaded via `using Package` without a corresponding project file - packages are also treated as stdlibs by Pkg - packages source code is not copied, shims are added so that Pkg will properly treat it as an stdlib
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1073 +/- ##
==========================================
- Coverage 86.21% 79.86% -6.35%
==========================================
Files 3 3
Lines 805 869 +64
==========================================
Hits 694 694
- Misses 111 175 +64 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
| - `copy_globs::Vector{String}=String[]`: Glob patterns for copying package files to the stdlib directory. | ||
| Patterns are relative to each package root and apply to all packages in the distribution. | ||
| Example: `["assets/**", "data/**"]` copies assets and data directories for all packages. |
There was a problem hiding this comment.
When specifying globs, if one forgets to include "src/**", you end up with a distribution that does not work because none of the packages can be loaded. I'm not 100% sure if this is true in general, but in my testing when I messed this up nothing would work.
I'm wondering if making the stub generation conditional on whether the src folder was copied over. That way there is no extra configuration to be done, although I guess there could be an argument to explicitly disable it, and when someone forgets to add "src/**" to the globs the distribution still ends up working.
There was a problem hiding this comment.
Perhaps the stub generation is even conditional on whether the package entrypoint is already in the target location or not.
There was a problem hiding this comment.
I don't think this is true unless the package somehow reads file during runtime from src. Otherwise, packages should be in the sysimage and not require any bundled sources. Do you have an example of this with some simple packages (Example.jl or Plots.jl etc?).
|
Just noting that on
Extensions are treated badly, as the code for the packages in the distribution are lost, unless they are copied over with |
|
Could we also customize the Basically this text: 
|
| - `include_transitive_dependencies::Bool=true`: If `true`, include transitive dependencies in the sysimage. | ||
| - `include_preferences::Bool=true`: Bundle package preferences into `share/julia/LocalPreferences.toml`. | ||
| - `script::Union{Nothing,String}=nothing`: Optional script executed while generating the sysimage. | ||
| - `copy_globs::Vector{String}=String[]`: Glob patterns for copying package files to the stdlib directory. |
There was a problem hiding this comment.
Could we extend this to allow an array of anything that glob accepts as a pattern, i.e. "strings, GlobMatches, a vector of strings or objects which implement occursin, including RegEx and Glob.FilenameMatch objects"?
In particular, I have the case where I need case insensitive matching which as far as I can tell I cannot express using a regular string converted into a GlobMatch under the hood. Writing all the possible scenarios out is also a bit cumbersome/noisy.
If the array/fn inputs for glob would be supported the following would be able to find both files with a single matcher.
julia> glob("license*", "$(homedir())/.julia/packages/ModelingToolkit/5RGAg/")
String[]
julia> glob([fn"license*"i], "$(homedir())/.julia/packages/ModelingToolkit/5RGAg/")
1-element Vector{String}:
"~/.julia/packages/ModelingToolkit/5RGAg/LICENSE.md"
julia> glob("license*", "$(homedir())/.julia/packages/OrderedCollections/Xihhq/")
String[]
julia> glob([fn"license*"i], "$(homedir())/.julia/packages/OrderedCollections/Xihhq/")
1-element Vector{String}:
"~/.julia/packages/OrderedCollections/Xihhq/License.md"
|
|
|
TL;DR: Few packages depend on files outside These packages fail to precompile without following files: This blocks building any sysimage with |
4 participants
This creates basically a new julia "distribution" with a custom set of packages in it.
Right now:
using Packagewithout a corresponding project fileTODO: