librealsense

re work) https://github.com/IntelRealSense/librealsense/pull/5586 https://sourceforge.net/p/speed-dreams/tickets/1022/ For Rust programs, many packages used i8 for C strings (to interface with OpenGL for instance) which broke when running on ppc64 and had to be changed to c_char, but that’s not a problem these days, I guess because many people now test on ARM. https://github.com/femtovg/femtovg/pull/5 Before Power10 was done, IBM actually asked us Raptor users about proposals for useful machine code instructions to add to it. I replied that I’d like to have hardware UTF-8 de-/encoding but they wanted a more detailed proposal and I never got around to write it. I’m not even sure that this would be worthwhile, but I see UTF-8 de-/encoding everywhere in the code I write and would like it to approach memory read/write speeds. I was very disappointed to learn that they had gone more proprietary with Power10 so I would not have been able to use those instructions anyway. What a pity! This machine still covers my needs so I’m not planning to replace it any time soon.

for using an unsupported environment and expecting it will work. Software in general is broken. It's amazing that it works as well as it does.

processes handling the spatial data and feeding extracted relevant contextual information to a wireless headset at a pretty crazy rate. Just to give you an idea a SLAM[4] point cloud from a stereo set of cameras or a cloud from a Kinect2 or Realsese produces a stream of data that is about 200mb a second. Google Draco[5] can compress that and help you stream that data at 1/8 the size without any tuning. Extracting skeletal information from that is really something that only Microsoft has reliably managed to deliver and it's at the core of the Kinect/Hololens products. NuiTrack[6] is the next best thing, but registering a human involves a T-Pose and gets tricky. Definitely you could roll something specific to the application, maybe just put a fiduciary marker[7] or two on a person and extrapolate skeleton from knowing where it is on their shirt. You will also want to be streaming back the RGBd, IMU, hand and skeletal tracking from the headset back to the server. This could help inform and reduce the tracking requirements from the surrounding sensors. Out of the box, you'd probably need a base i7, 64Gb+ of ram, and a couple GTX 1080s to power 4 sensors in one room. The task of syncing the cameras and orienting[8] them would be something you'd have to solve independently. After having all of that, you would have an amazing lab to reduce the problem further and maybe require less bandwidth, but very probably to get where you're going you'd need to scale that up by 2x for dev headroom and maybe run some sort of cluster operations[9] for management of your GPU processes and pipeline. Keeping everything applicable in memory for transport+processing would be desirable so you'd want to look at something like Apache Arrow[10]. At this point you are on the edge of what is possible at even the best labs at Google, Microsoft, or Apple. The arrow people will gladly welcome you as a contributor! Hope you like data science and vector math, because that's where you live now. After getting all of this orchestrated, you now have to stream an appropriate networked "game" environment[11] to your application client on the hololens, but congrats! You made a baby step! Battery life is still an issue, but Disney Research has demonstrated full room wireless power[12]. Now all you have to do is figure out all the UI/UX, gesture control, text entry/speech recognition, art assets, textures, models, particle effects, post processing pipelines, spatial audio systems[13], internal APIs, cloud service APIs, and application build pipeline. The Unity asset store has a ton of that stuff, so you don't have to get in the weeds making it yourself but you will probably have to do a big lift on getting your XR/Holodeck cluster processing pipeline to produce the things you want as instantiated game objects. Once that's done, you literally have a reality warping metaverse overlay platform to help people find their car keys. What's crazy is that you can probably have all of it for under $15,000 in gear. Getting it to work right is where the prizes are currently living and they are huge prizes. [0] https://en.wikipedia.org/wiki/ImageNet [1] https://azure.microsoft.com/en-us/services/kinect-dk/ [2] https://github.com/IntelRealSense/librealsense [3] https://www.vive.com/us/vive-tracker/ [4] http://webdiis.unizar.es/~jcivera/papers/pire_etal_iros15.pd... [5] https://github.com/google/draco [6] https://nuitrack.com/ [7] https://www.youtube.com/watch?v=JzlsvFN_5HI (markers are on the boxes not the robot) [8] https://realsense.intel.com/wp-content/uploads/sites/63/Mult... [9] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus... [10] https://arrow.apache.org/ [11] https://docs.unity3d.com/Manual/UNetClientServer.html [12] https://www.youtube.com/watch?v=gn7T599QaN8 [13] https://en.wikipedia.org/wiki/Ambisonics