Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [......XXXX..XXX.XX...X.X....X..X.X....X.......X.............] duration: 14.301171691s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:08:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 14.301205163s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................X.................] duration: 255.843845ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:07:45 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 255.885185ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.......XXXX..XXX.XX...X.X....X..X.X....X.......X............] duration: 5.740965073s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:07:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.740995581s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [X..............X............................................] duration: 14.84377601s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:07:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 14.843827119s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [........XXXX..XXX.XX...X.X....X..X.X....X.......X...........] duration: 5.956933979s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:07:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.956965992s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X..............X...........................................] duration: 29.506727546s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:06:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 29.506752234s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X........XXXX..XXX.XX...X.X....X..X.X....X.......X..........] duration: 7.280499076s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:06:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 7.280536747s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................X................] duration: 251.564037ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:06:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.59134ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X........XXXX..XXX.XX...X.X....X..X.X....X.......X.........] duration: 10.802534938s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 10.802553914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X.X........XXXX..XXX.XX...X.X....X..X.X....X.......X........] duration: 8.582763961s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:05:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 8.582801442s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X..............X..........................................] duration: 8.92047291s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:05:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 8.920499983s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [............................................X...............] duration: 251.161832ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:05:28 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.185155ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...X..............X.........................................] duration: 25.457377616s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:05:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 25.457407172s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.............................................X..............] duration: 252.562849ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:05:06 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 252.58956ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [XX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X.......] duration: 6.26330876s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:05:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.263344198s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..............................................X.............] duration: 249.451032ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:04:38 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 249.477663ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [XXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X......] duration: 15.833913558s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:04:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 15.833942021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X..............X........................................] duration: 15.464091019s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:04:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.464118712s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...............................................X............] duration: 250.571087ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:04:05 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 250.595332ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X..............X.......................................] duration: 15.376804001s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:03:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.376896577s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [................................................X...........] duration: 251.66567ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:03:33 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.774437ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X.....] duration: 5.956145189s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:03:20 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 5.956171919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X..............X......................................] duration: 35.184954184s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:03:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 35.184983289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X....] duration: 6.022011918s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:02:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.022038881s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.................................................X..........] duration: 251.885129ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:02:26 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 251.912821ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X...] duration: 16.399348911s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:02:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 16.39937495s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X..............X.....................................] duration: 15.324718358s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:01:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.324934039s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..................................................X.........] duration: 258.39715ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:01:51 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 258.424673ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X..] duration: 16.342730852s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:01:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 16.342758935s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X..............X....................................] duration: 15.400798945s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.40082707s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...................................................X........] duration: 254.593977ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:01:09 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.623462ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X.] duration: 6.082465151s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 6.08249557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X..............X...................................] duration: 15.804951417s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.804980953s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [....................................................X.......] duration: 240.96758ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 20:00:39 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 240.991595ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X..............X..................................] duration: 15.350430371s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 20:00:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.350460809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......X] duration: 30.003070245s trace: https://jaeger.dev.storj.io/trace/2650aae6847a0dcd FAIL (2025-12-29 20:00:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q) context deadline exceeded ======LOGS====== Dec 29 19:59:30.785682342 (580h30m30.656765377s) [upload=508876]: Starting upload Dec 29 19:59:30.785737928 (580h30m30.656820963s) [upload=508876]: Done starting upload Dec 29 19:59:30.785822799 (580h30m30.656905834s) [upload=508876]: Uploading segments... Dec 29 19:59:30.785935044 (580h30m30.657018079s) [upload=508876]: Got next segment. Inline: false Dec 29 19:59:30.785961944 (580h30m30.657044979s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:59:30.785966504 (580h30m30.657049549s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:59:31.037322625 (580h30m30.90840567s) [seg_pos={0 0}, upload=508876]: Begin upload segment... Dec 29 19:59:31.037376888 (580h30m30.908459923s) [seg_pos={0 0}, upload=508876]: Done begin upload segment. Dec 29 19:59:31.037410943 (580h30m30.908493978s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.037581418 (580h30m30.908664493s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.038394326 (580h30m30.909477371s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039164643 (580h30m30.910247678s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039271116 (580h30m30.910354161s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.039849837 (580h30m30.910932872s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040383864 (580h30m30.911466909s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.040931737 (580h30m30.912014782s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.041497955 (580h30m30.912581s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042074924 (580h30m30.913157959s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042144016 (580h30m30.913227051s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042641633 (580h30m30.913724678s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.042812168 (580h30m30.913895203s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.043486091 (580h30m30.914569136s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.044156338 (580h30m30.915239383s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:31.045260009 (580h30m30.916343044s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Uploading piece... Dec 29 19:59:32.743061497 (580h30m32.614144532s) [upload=508876]: Next returned nil segment Dec 29 19:59:32.743074252 (580h30m32.614157287s) [upload=508876]: Waiting for error group managing segments... Dec 29 19:59:33.017502537 (580h30m32.888585592s) [noise=true , address=145.79.227.38:4977 , piece_id=RWVYRKAYWV7T5CQKHFBPHEBLB342CBAOVHD6CURLLWO7WU3ZINXQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.025554098 (580h30m32.896637133s) [noise=true , address=145.79.227.13:4804 , piece_id=CFZB23OLJAOWRVJ7HN7GGMBNNOZKJCYQLM3Q466ZA6XAEBYY4MRQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.030515044 (580h30m32.901598079s) [noise=true , address=145.79.227.11:4731 , piece_id=PELAN445WQCZQXWWABGIZXME3VUBBN4VJ2LBHQSDLN6LSMME5YWQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.039091876 (580h30m32.910174911s) [noise=true , address=145.79.227.8:4329 , piece_id=EDA2IZIKEZ6WDFZ7EZ7NJIIHXROMT3NIOI3IJFMS6CJOHBZP7YPQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.044778073 (580h30m32.915861109s) [noise=true , address=145.79.227.16:4909 , piece_id=XIVWCR6ABV5XOUSBUKK2DADA3FMLDDVBJXIX23OONUZ6FIBB6QVA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.050402793 (580h30m32.921485828s) [noise=true , address=145.79.227.5:4221 , piece_id=C66XPFNMAJPOI3EEROFIK732ADUNBF25M2RANADDIM5WRZ6ZSFCA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.212468852 (580h30m33.083551898s) [noise=true , address=77.90.31.18:4805 , piece_id=V5XYHDIP7NKRFMEEVHZJ4QGRJGGR2DJC6YSNLGARRI6GAFG7OZ7Q, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.314338478 (580h30m33.185421513s) [noise=true , address=77.90.31.16:4736 , piece_id=YEO56LM7KSSJCCUSWINK3O7RNVQ6CS4IRZHY2FZ6PFQTPFVG6PZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.316148305 (580h30m33.18723134s) [noise=true , address=77.90.31.14:4662 , piece_id=4DQ3V6UBVFMRIUASDPYBL63IB63SSPBV22MEOOJHZKQF3UUF6P4A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.327220057 (580h30m33.198303092s) [noise=true , address=45.91.116.14:4190 , piece_id=VMEGE3LQB6PDMBKNNDKKPUDZTZROYYZG3MAIDVVRKV5HXS33ROZQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.330821635 (580h30m33.20190467s) [noise=true , address=45.91.116.8:4045 , piece_id=EWRI3VVLRYCPMRTP3F5UJS6PVHL6DFEIXL252C6BXJGYLNPVJCMA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.439752295 (580h30m33.31083533s) [noise=true , address=145.79.227.14:4837 , piece_id=KPLKA2E3OVMEW3Y5YT6GFGC7STCFQRH4C3INC563USYRQSVNEA3A, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514625476 (580h30m33.385708511s) [noise=true , address=77.90.31.19:4842 , piece_id=V5V6LWKG5CNY5H5YYLYOQJ7QGB5YBORP7JJOEDCWZOSRNVMEVSNA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: <nil> Dec 29 19:59:33.514653188 (580h30m33.385736223s) [seg_pos={0 0}, upload=508876]: Segment reached optimal threshold of 13 pieces. Dec 29 19:59:33.514847208 (580h30m33.385930243s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.514854461 (580h30m33.385937496s) [noise=true , address=45.91.116.7:4010 , piece_id=2JM6F6X3DKEDZGZLUWPLSXVTU6ZZWWI7LN5OPDJDOVOUANJMNNYA, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515083207 (580h30m33.386166242s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515099438 (580h30m33.386182483s) [noise=true , address=145.79.227.49:4339 , piece_id=AZZVJEVOLNEMWUH6ZUN24Z3FNONNEOA4AQHL4DFN3TTWILJQXWUQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515194038 (580h30m33.386277083s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 29 19:59:33.515204027 (580h30m33.386287072s) [noise=true , address=145.79.227.35:4483 , piece_id=U74GFKIPXEFCUSUJHAOWRANHHMZEV2H53UISPVXQJXVQTY3H5HIQ, seg_pos={0 0}, upload=508876]: Long tail cancellation due to optimal threshold Dec 29 19:59:33.515260365 (580h30m33.38634341s) [seg_pos={0 0}, upload=508876]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:59:33.515420268 (580h30m33.386503324s) [upload=508876]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:59:33.515428134 (580h30m33.386511179s) [upload=508876]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:59:33.822524344 (580h30m33.693607399s) [upload=508876]: Done uploading segments... Dec 29 19:59:33.822549562 (580h30m33.693632627s) [upload=508876]: Upload finished. err: <nil> Duration: 30.003099161s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.....................................................X......] duration: 274.76216ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:59:46 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 274.79378ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X..............X.................................] duration: 15.450527815s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 19:59:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.450559606s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X..............X................................] duration: 15.405253418s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 19:59:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 15.405284267s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [......................................................X.....] duration: 234.364526ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:59:10 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 234.396427ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.......................................................X....] duration: 238.183357ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:58:36 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 238.207553ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.X..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.......] duration: 6.952338034s trace: https://jaeger.dev.storj.io/trace/3f150239c6998166 FAIL (2025-12-29 19:58:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ) context deadline exceeded ======LOGS====== Dec 29 19:52:51.308730300 (580h23m51.179813355s) [upload=508782]: Starting upload Dec 29 19:52:51.308804702 (580h23m51.179887757s) [upload=508782]: Done starting upload Dec 29 19:52:51.309024650 (580h23m51.180107695s) [upload=508782]: Uploading segments... Dec 29 19:52:51.309069507 (580h23m51.180152552s) [upload=508782]: Got next segment. Inline: false Dec 29 19:52:51.309100404 (580h23m51.18018345s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:52:51.309105455 (580h23m51.18018849s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:52:51.550802493 (580h23m51.421885528s) [seg_pos={0 0}, upload=508782]: Begin upload segment... Dec 29 19:52:51.550985641 (580h23m51.422068686s) [seg_pos={0 0}, upload=508782]: Done begin upload segment. Dec 29 19:52:51.551050405 (580h23m51.42213344s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551062998 (580h23m51.422146043s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551624698 (580h23m51.422707743s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.552531264 (580h23m51.423614309s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553008232 (580h23m51.424091267s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553452158 (580h23m51.424535193s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553975314 (580h23m51.425058359s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555025955 (580h23m51.426109s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555661374 (580h23m51.426744419s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.556287206 (580h23m51.427370241s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557268995 (580h23m51.42835204s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557505356 (580h23m51.428588391s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.559435291 (580h23m51.430518336s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.561092656 (580h23m51.432175691s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.562704296 (580h23m51.433787341s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.564241182 (580h23m51.435324227s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:53.252683109 (580h23m53.123766144s) [upload=508782]: Next returned nil segment Dec 29 19:52:53.252699460 (580h23m53.123782495s) [upload=508782]: Waiting for error group managing segments... Dec 29 19:52:53.548902896 (580h23m53.419985941s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.553093956 (580h23m53.424177001s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.555263757 (580h23m53.426346842s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.560853782 (580h23m53.431936817s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.564223298 (580h23m53.435306344s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.565698838 (580h23m53.436781883s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576075246 (580h23m53.447158291s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576724301 (580h23m53.447807346s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.590105642 (580h23m53.461188687s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.715418666 (580h23m53.586501711s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.741074930 (580h23m53.612157975s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.797631653 (580h23m53.668714698s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.832989710 (580h23m53.704072745s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.833010209 (580h23m53.704093244s) [seg_pos={0 0}, upload=508782]: Segment reached optimal threshold of 13 pieces. Dec 29 19:52:53.833210050 (580h23m53.704293095s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833216652 (580h23m53.704299697s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833263081 (580h23m53.704346116s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833271507 (580h23m53.704354542s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833328135 (580h23m53.70441117s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection Dec 29 19:52:53.833333776 (580h23m53.704416811s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833350347 (580h23m53.704433382s) [seg_pos={0 0}, upload=508782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:52:53.833419619 (580h23m53.704502664s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:52:53.833424458 (580h23m53.704507493s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:52:54.123488002 (580h23m53.994571047s) [upload=508782]: Done uploading segments... Dec 29 19:52:54.123515133 (580h23m53.994598178s) [upload=508782]: Upload finished. err: <nil> Duration: 6.952360716s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X..............X...............................] duration: 8.033395585s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 19:58:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 8.033424181s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.X..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X......] duration: 6.400307783s trace: https://jaeger.dev.storj.io/trace/3f150239c6998166 FAIL (2025-12-29 19:58:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ) context deadline exceeded ======LOGS====== Dec 29 19:52:51.308730300 (580h23m51.179813355s) [upload=508782]: Starting upload Dec 29 19:52:51.308804702 (580h23m51.179887757s) [upload=508782]: Done starting upload Dec 29 19:52:51.309024650 (580h23m51.180107695s) [upload=508782]: Uploading segments... Dec 29 19:52:51.309069507 (580h23m51.180152552s) [upload=508782]: Got next segment. Inline: false Dec 29 19:52:51.309100404 (580h23m51.18018345s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:52:51.309105455 (580h23m51.18018849s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:52:51.550802493 (580h23m51.421885528s) [seg_pos={0 0}, upload=508782]: Begin upload segment... Dec 29 19:52:51.550985641 (580h23m51.422068686s) [seg_pos={0 0}, upload=508782]: Done begin upload segment. Dec 29 19:52:51.551050405 (580h23m51.42213344s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551062998 (580h23m51.422146043s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551624698 (580h23m51.422707743s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.552531264 (580h23m51.423614309s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553008232 (580h23m51.424091267s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553452158 (580h23m51.424535193s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553975314 (580h23m51.425058359s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555025955 (580h23m51.426109s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555661374 (580h23m51.426744419s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.556287206 (580h23m51.427370241s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557268995 (580h23m51.42835204s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557505356 (580h23m51.428588391s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.559435291 (580h23m51.430518336s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.561092656 (580h23m51.432175691s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.562704296 (580h23m51.433787341s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.564241182 (580h23m51.435324227s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:53.252683109 (580h23m53.123766144s) [upload=508782]: Next returned nil segment Dec 29 19:52:53.252699460 (580h23m53.123782495s) [upload=508782]: Waiting for error group managing segments... Dec 29 19:52:53.548902896 (580h23m53.419985941s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.553093956 (580h23m53.424177001s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.555263757 (580h23m53.426346842s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.560853782 (580h23m53.431936817s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.564223298 (580h23m53.435306344s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.565698838 (580h23m53.436781883s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576075246 (580h23m53.447158291s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576724301 (580h23m53.447807346s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.590105642 (580h23m53.461188687s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.715418666 (580h23m53.586501711s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.741074930 (580h23m53.612157975s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.797631653 (580h23m53.668714698s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.832989710 (580h23m53.704072745s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.833010209 (580h23m53.704093244s) [seg_pos={0 0}, upload=508782]: Segment reached optimal threshold of 13 pieces. Dec 29 19:52:53.833210050 (580h23m53.704293095s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833216652 (580h23m53.704299697s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833263081 (580h23m53.704346116s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833271507 (580h23m53.704354542s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833328135 (580h23m53.70441117s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection Dec 29 19:52:53.833333776 (580h23m53.704416811s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833350347 (580h23m53.704433382s) [seg_pos={0 0}, upload=508782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:52:53.833419619 (580h23m53.704502664s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:52:53.833424458 (580h23m53.704507493s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:52:54.123488002 (580h23m53.994571047s) [upload=508782]: Done uploading segments... Dec 29 19:52:54.123515133 (580h23m53.994598178s) [upload=508782]: Upload finished. err: <nil> Duration: 6.400342679s core uplink us-select-1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [........................................................X...] duration: 241.865578ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:57:49 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 241.897671ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [.........................................................X..] duration: 254.108041ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:57:27 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 254.13435ms edge linksharing bgp eu1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.X..XXXXX.X........XXXX..XXX.XX...X.X....X..X.X....X.....] duration: 6.628423277s trace: https://jaeger.dev.storj.io/trace/3f150239c6998166 FAIL (2025-12-29 19:57:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ) context deadline exceeded ======LOGS====== Dec 29 19:52:51.308730300 (580h23m51.179813355s) [upload=508782]: Starting upload Dec 29 19:52:51.308804702 (580h23m51.179887757s) [upload=508782]: Done starting upload Dec 29 19:52:51.309024650 (580h23m51.180107695s) [upload=508782]: Uploading segments... Dec 29 19:52:51.309069507 (580h23m51.180152552s) [upload=508782]: Got next segment. Inline: false Dec 29 19:52:51.309100404 (580h23m51.18018345s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 29 19:52:51.309105455 (580h23m51.18018849s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 29 19:52:51.550802493 (580h23m51.421885528s) [seg_pos={0 0}, upload=508782]: Begin upload segment... Dec 29 19:52:51.550985641 (580h23m51.422068686s) [seg_pos={0 0}, upload=508782]: Done begin upload segment. Dec 29 19:52:51.551050405 (580h23m51.42213344s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551062998 (580h23m51.422146043s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.551624698 (580h23m51.422707743s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.552531264 (580h23m51.423614309s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553008232 (580h23m51.424091267s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553452158 (580h23m51.424535193s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.553975314 (580h23m51.425058359s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555025955 (580h23m51.426109s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.555661374 (580h23m51.426744419s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.556287206 (580h23m51.427370241s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557268995 (580h23m51.42835204s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.557505356 (580h23m51.428588391s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.559435291 (580h23m51.430518336s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.561092656 (580h23m51.432175691s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.562704296 (580h23m51.433787341s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:51.564241182 (580h23m51.435324227s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Uploading piece... Dec 29 19:52:53.252683109 (580h23m53.123766144s) [upload=508782]: Next returned nil segment Dec 29 19:52:53.252699460 (580h23m53.123782495s) [upload=508782]: Waiting for error group managing segments... Dec 29 19:52:53.548902896 (580h23m53.419985941s) [noise=true , address=145.79.227.10:4402 , piece_id=S7XRW77WX3Q7JKLU2ZGUV7NBNZJRU7XIGI5WMIBUIOTV5OVLI7GA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.553093956 (580h23m53.424177001s) [noise=true , address=145.79.227.45:4231 , piece_id=KCFBGNRDC7QWJMVL2RH4VK5XSG2M43G2SXXIFNOQLVAPUT2B7E5Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.555263757 (580h23m53.426346842s) [noise=true , address=145.79.227.24:4583 , piece_id=VQ3HF74PALHG6O34WAUW5XOYDVNVDS4WMOCWV5ZK572LK4LIQAWQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.560853782 (580h23m53.431936817s) [noise=true , address=145.79.227.13:4802 , piece_id=GDQJI552NIUWIRYTAEM5RAMV2IEGVC7XIODQAMHLV4S365OP6AEQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.564223298 (580h23m53.435306344s) [noise=true , address=145.79.227.42:4123 , piece_id=IMRZMTXBKN5SIPM2XENGONW4SO3SG2GN6RWMHUOOH7AOHRJ22Z7Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.565698838 (580h23m53.436781883s) [noise=true , address=145.79.227.6:4257 , piece_id=PWR25PZSVQGFIG3PEJEXP2TPPJYTQ6X2OPPFVVS4KPU4Y3Q5E6NA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576075246 (580h23m53.447158291s) [noise=true , address=145.79.227.3:4151 , piece_id=X5UWCFBV4XBUNB2Z5CEY3ENJDO2ULSMC3YHMLBDSVRUB4BXVPJKQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.576724301 (580h23m53.447807346s) [noise=true , address=145.79.227.32:4836 , piece_id=OQ3C2UCUZHL7RVD4FS4OVEWYMJF6MYXUA64JTM5WAQDDFS3Z3CLA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.590105642 (580h23m53.461188687s) [noise=true , address=145.79.227.47:4265 , piece_id=7O3PH427RIQPHOCAXPRUJ2CUX3I2ZXFB72R62LEOXCHM55DHNHXA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.715418666 (580h23m53.586501711s) [noise=true , address=77.90.31.18:4805 , piece_id=ROYUG6WZEGZYCTIYTTUINGSWVI5G7MSDX5WJHASL5G5TRTXNHZ2Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.741074930 (580h23m53.612157975s) [noise=true , address=77.90.31.19:4842 , piece_id=MBVDMUDSPAWRIORRJS6NIQP4WSRFBYDN6CFJXV46CW7O54BOV3YQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.797631653 (580h23m53.668714698s) [noise=true , address=77.90.31.12:4590 , piece_id=ZTIIID4Z224BSZY2G6XJQ5YFO4JIX3Z5PKMRDN3X7FD6FCYN2AQA, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.832989710 (580h23m53.704072745s) [noise=true , address=45.91.116.6:4976 , piece_id=ZBNQ65ERTMHJ4CEV2TTJMUPEDAZWC3ESPARUTHUEKKABHT4UMD3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: <nil> Dec 29 19:52:53.833010209 (580h23m53.704093244s) [seg_pos={0 0}, upload=508782]: Segment reached optimal threshold of 13 pieces. Dec 29 19:52:53.833210050 (580h23m53.704293095s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833216652 (580h23m53.704299697s) [noise=true , address=45.91.116.14:4189 , piece_id=SLTLVLE2S2AO5GINUGAJVW3DLS6WYHX6CPCLIUMOXY63OPQYPS7A, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833263081 (580h23m53.704346116s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 29 19:52:53.833271507 (580h23m53.704354542s) [noise=true , address=145.79.227.33:4869 , piece_id=L3IBDBCCLCQHTZF7IWUNRJUJKVHT4SHNTK7S2PCYPNFGL5AQATYQ, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833328135 (580h23m53.70441117s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50174->45.91.116.9:4263: use of closed network connection Dec 29 19:52:53.833333776 (580h23m53.704416811s) [noise=true , address=45.91.116.9:4263 , piece_id=IHTD6JIWA36F4DVTCHEX5TM7KYRSILEIXFQ6UO77LPKDJSPR6O3Q, seg_pos={0 0}, upload=508782]: Long tail cancellation due to optimal threshold Dec 29 19:52:53.833350347 (580h23m53.704433382s) [seg_pos={0 0}, upload=508782]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 29 19:52:53.833419619 (580h23m53.704502664s) [upload=508782]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:52:53.833424458 (580h23m53.704507493s) [upload=508782]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:52:54.123488002 (580h23m53.994571047s) [upload=508782]: Done uploading segments... Dec 29 19:52:54.123515133 (580h23m53.994598178s) [upload=508782]: Upload finished. err: <nil> Duration: 6.628449968s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X..............X..............................] duration: 32.102129334s trace: https://jaeger.dev.storj.io/trace/24ac2b258db6f49 FAIL (2025-12-29 19:57:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 19.881883s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.011913035 (580h4m19.88299608s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012333796 (580h4m19.883416831s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.012776579 (580h4m19.883859614s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013135943 (580h4m19.884218978s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013460732 (580h4m19.884543767s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.013853229 (580h4m19.884936264s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014144343 (580h4m19.885227388s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014250846 (580h4m19.885333881s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.014906585 (580h4m19.88598962s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.015511426 (580h4m19.886594471s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.016659873 (580h4m19.887742918s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.018312429 (580h4m19.889395464s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.019851220 (580h4m19.890934255s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:20.021306711 (580h4m19.892389746s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Uploading piece... Dec 29 19:33:21.467852993 (580h4m21.338936028s) [upload=508498]: Next returned nil segment Dec 29 19:33:21.467865737 (580h4m21.338948772s) [upload=508498]: Waiting for error group managing segments... Dec 29 19:33:21.864176322 (580h4m21.735259357s) [noise=true , address=145.79.227.11:4748 , piece_id=3FFT5U7HDYRGKI67WTT3EWL6PSQY4MOFM2RVALUI2SXDQVFAUV7A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.890454300 (580h4m21.761537345s) [noise=true , address=145.79.227.8:4335 , piece_id=Q7UEGDD4AGZREL2V2VRHEVDJFIDPKKUF6JCZ2FJGQYJAVW7KQAKQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.895347667 (580h4m21.766430713s) [noise=true , address=145.79.227.42:4150 , piece_id=TT4PDRJ6HRBNFYKWOBEHOPYSPMYVN2NZZKUBK3E5ACHRKJPK4ZMA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.912551226 (580h4m21.783634261s) [noise=true , address=145.79.227.29:4959 , piece_id=IUPWCJOL4JXCAOUTP4GCQMK657U53VGQD4JXRNNHSYLK2AK56WNA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:21.918776128 (580h4m21.789859173s) [noise=true , address=145.79.227.6:4283 , piece_id=UZISQ26DIRCHFWIKODJCEQYD2N3FEWFKC7CAL7LRB45KAGPHC5BQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.049639643 (580h4m21.920722678s) [noise=true , address=5.231.73.4:4027 , piece_id=W56MVTNRZX4XFEMCJULZEX6GXQ4BNDNJ3R3EZ5LDHWVCGGMB2NTQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.062422904 (580h4m21.933505939s) [noise=true , address=77.90.31.17:4786 , piece_id=EJTLQD7JFIXVQ4GZ6S35KOISBSCOVOCW44R7JDDN5HLWRXMZDQ6A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.087400355 (580h4m21.9584834s) [noise=true , address=77.90.31.18:4836 , piece_id=ONCF7RXVICUL55HJ2AWB466YFRANSOIN4WKMH5VCURXP334CRG5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.123477030 (580h4m21.994560075s) [noise=true , address=77.90.31.10:4542 , piece_id=77QBOT4PZXNTMK7Z4SNPYVSJDNKZAJYKZIE2XJO2KOKYNFS6XN6Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.147097236 (580h4m22.018180271s) [noise=true , address=77.90.31.10:4539 , piece_id=HFHV4GSLPWU2NXE24SCA7XHPSQMHEKCTT7HOPBXJYQDN5XYQBUTA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.187114676 (580h4m22.058197721s) [noise=true , address=45.91.116.9:4283 , piece_id=W2IRSXBWJZBSADPBETO254LH5WH7RWHLP65FMYCW3QLTDZMWAJ2A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.755107038 (580h4m22.626190073s) [noise=true , address=145.79.227.43:4190 , piece_id=5O4HKBMCZZ5FWQCKO2B3NJVBV2ZHAVZCYRYRPHCURU5ZQ2SPIE5A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.779882074 (580h4m22.650965109s) [noise=true , address=145.79.227.30:4773 , piece_id=BAR4QMZ7OM74IIPLRGWAMEYZKIILXXQNRYJFBUHXV5YUWYD7X2YA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820831227 (580h4m22.691914262s) [noise=true , address=145.79.227.44:4213 , piece_id=47WPDBUPSA5QUYOBYSFQFQMIFZMJJTD3LBYL6J4WCM4G2LLYKRBQ, seg_pos={0 0}, upload=508498]: Done uploading piece. err: <nil> Dec 29 19:33:22.820854071 (580h4m22.691937106s) [seg_pos={0 0}, upload=508498]: Segment reached optimal threshold of 14 pieces. Dec 29 19:33:22.821045084 (580h4m22.692128119s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1228o9xxwySb7RCPH81yobfdRxNQoz82JYgkb9TEw8eeX7CATgC): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821051897 (580h4m22.692134932s) [noise=true , address=145.79.227.34:4913 , piece_id=O22AXRPCITP6BCWH4ZANG2Z2F4PENTRZOCIA3YUMXYBVP7XXRWUA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821168860 (580h4m22.692251905s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nxT1Ldr46HPrDRHQ2nrwEBZT5bEVoPAAfBz86ZYwUH13sqwHu): protocol: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52944->64.49.13.169:4711: use of closed network connection Dec 29 19:33:22.821176013 (580h4m22.692259049s) [noise=true , address=64.49.13.169:4711 , piece_id=MZYKDHA7E4KNOZ4C752DWBLJJPJQTBGCRX6QACQCCCIJ5VT4LU4Q, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821193216 (580h4m22.692276251s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gUV64TP8cTQhBjswnhcWfQHwYEEWqr5ASuPjtEx3X6ZQNVhS5): protocol: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47300->64.49.13.27:4966: use of closed network connection Dec 29 19:33:22.821198947 (580h4m22.692281983s) [noise=true , address=64.49.13.27:4966 , piece_id=GABX3GQG7TGY36WBSO3JWMBEIOK7R7MDZA4IHPPNBRBFXBTMJO3A, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821300189 (580h4m22.692383225s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17VMFZ2T8jrSJVtchciJ9SnyrR6JstvcMoXrN5Q3zTD1aQU1oJ): protocol: expected piece hash; context canceled; EOF Dec 29 19:33:22.821306302 (580h4m22.692389337s) [noise=true , address=145.79.227.33:4893 , piece_id=QTATPQWRYUQ33FL2M3RAP762AM3AC4PT7BHBCIUT7GFNVDC5UDGA, seg_pos={0 0}, upload=508498]: Long tail cancellation due to optimal threshold Dec 29 19:33:22.821399579 (580h4m22.692482614s) [seg_pos={0 0}, upload=508498]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 29 19:33:22.821465755 (580h4m22.6925488s) [upload=508498]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 29 19:33:22.821470243 (580h4m22.692553278s) [upload=508498]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 29 19:33:23.123995567 (580h4m22.995078612s) [upload=508498]: Done uploading segments... Dec 29 19:33:23.124020665 (580h4m22.99510371s) [upload=508498]: Upload finished. err: <nil> Duration: 32.102155445s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [..........................................................X.] duration: 236.720762ms remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:57:08 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 236.739076ms edge linksharing bgp eu1
checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 2m0.001189225s remote: "tcp:207.211.208.129:443" target: https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt trace: https://jaeger.dev.storj.io/trace/4679501fce62ca7 FAIL (2025-12-29 19:57:07 UTC) orbiter warning: Get "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt": context deadline exceeded Duration: 2m0.001211027s edge linksharing bgp eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)

checkers.HTTP{"GET" "https://link.storjshare.io/raw/jwhnbh22a7huxgm5mpnu45kaahuq/uptime-checks/hello-world.txt", ExpectedContent: "Hello, world!"} failure (antiblip window 6/60): (http.eu1.linksharing)