This topic contains 14 replies, has 3 voices, and was last updated by Jeremy Hill 5 months, 3 weeks ago.
September 4, 2019 at 11:10 am #2033
I’m confused by this, I presumed as per the downloads info for cli, that I can render scenes directly from the cli.
‘Bella CLI is a command-line program which can be used to implement your own render farm’
Using -i:bellarenderscene.bsx as an input path, tells me the cli doesn’t know what to do with the input?
So, am I correct in realising that the cli is purely for merging bsi files, post render on several machines?
How do we set, if it is actually necessary, the seed for each render machine?
Do I just render using the gui on each machine and then merge afterwards using the cli?
Sorry been tainted by other unbiased render engines, which have network rendering elements. :roll
Tim.September 4, 2019 at 1:23 pm #2034
No, the cli for rendering too, It should begin rendering bellarenderscene.bsx — I just double-checked here on win & ubuntu and see no issue. To reach the error you indicate, it looks like the given path must not exist, so I’m not sure what may be the issue (case sensitivity, in case of linux?). But I think the reporting could be improved, so I’ll do that.
Regarding seed, if I recall correctly, Albert has implemented it such that this (setting it manually) should be unnecessary.September 4, 2019 at 1:31 pm #2035
Hey sir, I missed enclosing the file path in ” ” and think this what failed it, as the path had spaces in several folder names.
Do I add output for image and bsi as such?:-
bella_cli.exe -i:"Z:\08 SHARED\SOFTWARE FOR PCS\bellarender\scenes\SWC\SingleWavelengthCornell.bsx" -o:C:\Users\Mal.Render\Documents\bellarender\renders\SWC_001.png -o:C:\Users\Mal.Render\Documents\bellarender\renders\SWC_V001.bsi
Z:\ is mapped so didn’t think I’d need UNC filepaths, but can switch this out if need be. Have a meeting now but will check cli afterwards with ” file path “.
Tim.September 4, 2019 at 1:44 pm #2036
Yep the seed is unnecessary, Bella has been designed in a way that you have not to worry about selecting a seed, as is virtually impossible for any computer to repeat the render sequence from any other. You can do both with CLI, render and merge. Just to mention, be sure to merge them once only, as I think there is no detection at this point for same file to be merged multiple times.September 4, 2019 at 1:46 pm #2037
Up to this point I have really added no override functionality to the cli, so it will simply read the input file and render to the file’s natural output paths (i.e. according to the settings, and how each pass is set up). Of course we are free to pollute the cli with as many options as we wish, but I preferred to wait on that, to get concrete input from people like yourself who actually know how to run a farm.September 4, 2019 at 3:02 pm #2038
Ok Albert, thank you.
This is currently not going to work I’m sorry to say. Here are some issues as I see them.
1. Using the same ‘in scene’ file output on all rendernodes will all write their files to the same place.
2. bsi files from rendernodes will need incremental filenaming, so they don’t overwrite the same file
3. Farms tend to rely on UNC file paths which if locally set file paths are saved in the scene, ie mapped drives which are not mapped on the rendernodes, the output file saving will fail.
4.Currently I can only see farm rendering being able to work, if I render with GUI on each node and add _N1, _N2, _N3 etc for each rendernodes output, so bsi files are all saved to a UNC network location, without overwriting each other.
5. Any setting other than ‘Save at end’ for the updates, will kill network traffic, so locally saved bsi will be required with save to network location on render finishing. Save at end is a killer if power fails, machine crashes etc. occur before a render finishes. Updating a local file each SL, or as set in scene, will give recoverable render outputs.
6. Farm managers, such as Deadline, will require a specified output file path.
7. Collect dependencies and send with the scene, ie Pack & Go, will be required for non-mapped drives & network locations which may be used for textures. (Our farm does not have mapped drives, so everything has to be sent with the 3dsMax / Maya / VRay scenes when rendering, as I’m probably the only one here who knows the UNC filepath names lol.)
I will have a go with the current cli as is, but I think you will need to expose some overrides Jeremy, to get around my concerns above.
I built the help html from the cli, but couldn’t locate a bnd to get more info about the cli other than the /h -h option. Hence asking here, no issue at this stage in your dev process though.
Tim.September 4, 2019 at 4:04 pm #2041
Definitely I will add whatever is necessary, I just need input like you give above, before deciding exactly how to do things. One of my main goals with this is to avoid creating yet another network manager, and instead design the cli such that it integrates as seamlessly as possible with existing third-party tools.
The scene model is built such that changing just settings:outputDir is sufficient to redirect all outputs, provided they do not use explicit override paths. And by default, settings:outputDir is blank, such that output paths depend on the bsx path (btw I will generally use bsx to refer to any bella scene file, bsa, bsx, or bsz). So a first step will pretty clearly be an –outputDir cli option. And a second might be, in the specific case of cli rendering, to also provide some way of redirecting even explicitly-overriden output paths, though I would need to think about that.
Likewise, outputs by default are all dependent upon settings:outputName, such that an –outputName cli option should work, for instance, to name outputs from a given node, according to that node, for purposes of being able to predict what comes out of the renderer, in scripts, and so forth.
From there, you have questions about bsi, and from what you say, I infer that you like to dump N bsis from N nodes, into a common directory somewhere on the network. And that you require control over an interval, according to which local bsis are copied to that common dir.
Regarding pack & go, this is already implemented in the form of bsz. It is not yet exposed in maya, as I have not yet decided on a clean way of doing that, but from bella gui, it is as simple as choosing .bsz instead of .bsa/.bsx when you use Save As. Using a bsz should work transparently, whether in gui or cli.September 4, 2019 at 5:24 pm #2045
Thanks Tim for all that information. Regarding:
2. bsi files from rendernodes will need incremental filenaming, so they don’t overwrite the same file
It seems so you would need something like an “id” of each bsi to avoid being overwritten. Bella cannot know how many machines are going to be used cooperatively, without a network manager. Due to that, the “id” cannot either be incremental, as that would probably result into performance penalties, if each render node needs to search all existing bsi in the folder to try to make it incremental.
I see 3 possible solutions to this:
a ) user is fully responsible in network rendering to assign that different naming in the bsi, output, etc to not overwrite
b ) user tells an id-host to bella it means bella will use that id to write bsi/output with that id ( I think thats similar to give seed ), and we can use id-host 0 default to ignore that extra-id-naming
c ) user tells bella render is gonna be cooperative, and telling a hint of how many computers ( that might help in some already discussed scenarios, specially for Apollo solver ). Then Bella uses internal id generated to write the output with id. The problem with that is not gonna be incremental but looks something like output_A34B291B.bsi, output_B5342CD9.bsi, etc. We can add then an option in CLI to merge all bsi’s in a folder, so that way do not need to be incremental.
I cannot see further, I have a limited knowledge of render farm needs.September 4, 2019 at 5:31 pm #2046
I am currently implementing –outputDir and –outputName as described above. However, when it comes to the question of explicitly-overridden output paths, my question becomes: is it ever going to be desirable, when overriding –outputDir in cli, to have any pass writing to a directory other than –outputDir?
So far, I can’t see that it is, except with the possible exception that the pass override dir is relative, but even in that case, from the point of view of someone running a farm, that would just seem to make things confusing and difficult, with no apparent benefit. So at the moment, I am inclined to go through all passes and set any with explicit output paths to use –outputDir. Please feel free to weigh-in with any thoughts on this.
And a side note, I have checked to make sure that bella is working fine with either UNC paths, or mapped drives (though in the latter case there is a caveat: I am aware of no way to resolve a mapped drive if the current user is running as admin), at least here on my network (win10 cli rendering a bsz on a drive mapped to a folder on a mac, with –outputDir set to a nonexistent dir on the win10 machine, in this test).September 4, 2019 at 7:54 pm #2052
What I am considering for bsi is to add –finalBsiDir and –finalBsiName cli arguments, where a process will render locally as normal, and then at the end of rendering, copy the local bsi to the specified directory (if given), using the specified name (if given).
This allows final bsi naming to be completely determined by the client, who is therefore able to prevent collisions if desired. And if he does not do so (e.g. setting –finalBsiDir but not –finalBsiName), then we may end up with name collisions in the target dir, and here I would generate incremental names. I do not see performance figuring much here, as we are talking about a one-time thing, per node.
However I am not sure about what Albert mentions about apollo & cooperative, so I’ll need to consult with the guys.
(edit: nevermind on the last thing, he just meant it can be helpful to know the total number of machines, so that’s easy)September 4, 2019 at 8:46 pm #2053
Firstly please forgive my unfamiliarity with bella’s exposed and not yet exposed functions. I’m still finding my way around the software.
Thank you both for your replies and answers.
I now have a better grasp of what you intend to do and how you will implement it.
Regarding your answers:-
Bsi auto naming based on guid sounds ideal and paired with -outputdir: or finalbsidir etc for cli based rendering, I think this will be perfect.
No need for incremental numbering and unknowns within rendernode numbers etc.
Bsz for collecting dependencies is great too and I had not realised that’s it’s purpose..
No need for a network manager then, as you mentioned Jeremy and you can rely on 3rd party managers with these simple additions and bsz.
Deadline has a commandline plugin which can be adapted to suit bella_cli easily.
I have written my own Deadline plugin for a cli argument based version, of the image formatting app I wrote for my current workplace. Due to contractual workplace regulations I would unfortunately not be able to share it, but would be happy to work up a fresh version for bella_cli if you wish.
If local rendering can be done with cli and render to final output for merging, upon finishing, then that sounds ideal too.
Both my farm and my workplace farm are all admin user machines, so unc paths would be best too. Deadline documentation suggests using unc paths although they do have a setting in the main monitor app, to set up path mapping which is then forwarded to the slaves on the render nodes. Never used it though as I prefer unc for textures and input / output paths.
Great stuff and I cant wait to test it.
Tim.September 4, 2019 at 8:50 pm #2054
Having seen your edit addition, Mxs plugin for Deadline counts the machines specified as to work out combined SL and time to complete using Juan’s magic log algorithm.
If any mxis are missing before merge, you have the option to ignore the missing ones or wait.
So number of machines is not always a necessity, unless you want to send feedback progress for combined bsi SL and time remaing to complete the render.
Tim.September 4, 2019 at 9:25 pm #2057
As I understand it in the case of apollo, it (knowing the number of machines) can actually be beneficial to total render time. I was afraid it was going to be something like needing to have one node get access to bsi files from other nodes, but Albert says just knowing the number is enough for this particular optimization.
Regarding bsi auto-naming using guids, I think I’d leave that up to the user, via –finalBsiName. At least to start with. Probably I have not mentioned yet that when things are a bit more clear, I will add macro support for filenames and such, in which case a $guid macro would do the trick.September 4, 2019 at 9:44 pm #2058
Yes, with respect to guid, I was referring to Albert’s ‘C’ option in his post above, rather than an end user requiring them.
Sensible for a farm user to enter the number of render nodes for use, prior to launching a render as well.
Macros would be a good addition too.
👍👊September 4, 2019 at 11:40 pm #2086
Made builds with this new cli stuff, and linked them on the new builds page, where adventurous people can go to get their hands on the most recent code.
The way these cli additions are implemented is as described above:
–outputDir [-od] overrides settings:outputDir, and the dir of any passes with explicit override paths.
–outputName [-on] overrides settings:outputName.
–finalBsiDir [-fbd] causes bella to copy local bsi to the specified dir, upon successful render completion.
–finalBsiName [-fbn] overrides the natural bsi output name.
If, while copying bsi to the final bsi dir, there are name collisions, the (final) bsi name will be incremented with an integer suffix.
(edit: damn this wordpress auto-changing double dashes to a single long dash behavior — the full cli argument names above are supposed to be shown with two preceding dashes)
You must be logged in to reply to this topic.