Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X......................X.......] duration: 5.77925471s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:21:10 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.779285649s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................X...........................................] duration: 3.419983621s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:20:37 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.420026052s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.723408358s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:20:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.723605845s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X......................X......] duration: 5.98496393s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:20:06 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.984996093s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X......................X.....] duration: 5.58352234s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:19:34 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.583551395s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.................X..........................................] duration: 3.480348315s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:19:33 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.480378753s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.754503468s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:19:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.754532042s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X......................X....] duration: 5.723388269s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:19:13 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.723419537s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.85029885s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:19:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.850318868s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..................X.........................................] duration: 3.448760056s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:19:01 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.448886686s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X......................X...] duration: 5.932894607s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:18:46 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.932920657s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.726735783s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:18:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.726764638s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...................X........................................] duration: 3.392985517s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:18:02 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.393083665s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X......................X..] duration: 5.830023413s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:17:50 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.8300638s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....................X.......................................] duration: 3.462216772s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:17:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.462244224s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.773281636s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:17:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.773314577s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X......................X.] duration: 5.792023133s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:17:17 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 5.792055042s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....................X......................................] duration: 3.488783068s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:17:16 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.4888108s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.899774175s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:17:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.899901428s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......................X.....................................] duration: 3.319481665s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:16:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.319519257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.791799476s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:16:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.791829914s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X......................X] duration: 30.006937163s trace: https://jaeger.dev.storj.io/trace/577eef994ac84ba2 FAIL (2025-11-25 06:16:36 UTC) orbiter warning: operation: sync2.Copy uplink: metaclient: context deadline exceeded Duration: 30.006960758s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......................X....................................] duration: 3.666298276s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:16:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.66631559s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.731971375s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:15:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.732004978s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.992813814s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:15:49 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.99284278s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.021913086s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:15:08 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 6.021959474s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........................X...................................] duration: 3.624288133s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:15:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.6243135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.8749434s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:14:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.874971863s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........................X..................................] duration: 3.44038642s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:14:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.440416065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.845498029s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:14:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.845529809s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.795032554s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:14:05 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.795059965s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.941360087s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:13:20 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.941392619s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........................X.................................] duration: 3.392612931s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:13:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.392661272s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.752652455s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:13:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.752684857s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................X................................] duration: 3.45565863s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:12:46 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.455706391s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.593298666s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:12:22 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.593333512s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.677892188s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:12:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.677921123s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............................X...............................] duration: 4.505044872s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:11:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.505066884s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............................X..............................] duration: 3.467572927s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:11:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.467588197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.708570383s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:11:30 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.708599767s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.969595881s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:11:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.969626449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..............................X.............................] duration: 3.490510495s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:11:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.490542225s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.87331731s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:10:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.873367086s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.681026875s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:10:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.68105054s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.757810125s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:10:36 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.757825433s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.738233609s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:10:18 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.73826055s core uplink us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...............................X............................] duration: 3.555450933s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:10:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.555483455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.919476847s trace: https://jaeger.dev.storj.io/trace/5cf2ad862c20f71d FAIL (2025-11-25 06:09:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Nov 25 05:59:02.986103331 (381h55m1.537573792s) [upload=336349]: Starting upload Nov 25 05:59:02.986248708 (381h55m1.537719169s) [upload=336349]: Done starting upload Nov 25 05:59:02.986432077 (381h55m1.537902548s) [upload=336349]: Uploading segments... Nov 25 05:59:02.986588434 (381h55m1.538058895s) [upload=336349]: Got next segment. Inline: false Nov 25 05:59:02.986647146 (381h55m1.538117608s) [upload=336349]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 25 05:59:02.986655061 (381h55m1.538125522s) [upload=336349]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 25 05:59:16.850251252 (381h55m15.401721733s) [seg_pos={0 0}, upload=336349]: Begin upload segment... Nov 25 05:59:16.850304353 (381h55m15.401774814s) [seg_pos={0 0}, upload=336349]: Done begin upload segment. Nov 25 05:59:16.850335673 (381h55m15.401806134s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850396278 (381h55m15.401866729s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.850898234 (381h55m15.402368695s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851221810 (381h55m15.402692271s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851781436 (381h55m15.403251897s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.851964885 (381h55m15.403435346s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852103399 (381h55m15.40357386s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852400214 (381h55m15.403870665s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852555941 (381h55m15.404026402s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.852764548 (381h55m15.404235009s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853083515 (381h55m15.404553976s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853151805 (381h55m15.404622266s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853365442 (381h55m15.404835893s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.853636498 (381h55m15.405106959s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854046910 (381h55m15.405517371s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:16.854617326 (381h55m15.406087787s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Uploading piece... Nov 25 05:59:18.650439352 (381h55m17.201909813s) [upload=336349]: Next returned nil segment Nov 25 05:59:18.650453830 (381h55m17.201924281s) [upload=336349]: Waiting for error group managing segments... Nov 25 05:59:18.953671099 (381h55m17.50514156s) [noise=true , address=77.90.31.17:4772 , piece_id=FV5KNEWOIE64KKTKXAE23GAOPORH73RV7QJE3YMPIQHKRTU7RIZQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:18.986665647 (381h55m17.538136108s) [noise=true , address=77.90.31.19:4843 , piece_id=7NMOPJJE4KBF3UBZWEUXHZENRKSJ7BM2AONCBYKCRUUA67GMQEXQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.011227487 (381h55m17.562697948s) [noise=true , address=77.90.31.11:4555 , piece_id=G376EHKSPYJ6CHL7V5MVZ4XFT7MB7FQ5XDK2JQLUIWDB3QQ4QDTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.027391897 (381h55m17.578862358s) [noise=true , address=45.91.116.4:4937 , piece_id=R7J7A4FOQF6TLMUDOSH2HIDDUF5UOZNVQZZCXPDYRCL74GSBKPQQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:19.055370142 (381h55m17.606840593s) [noise=true , address=77.90.31.15:4698 , piece_id=B2HKELREKYI5MQ6FG2DQXWIUSOLY2HSWUILF3UL7CXUFDGY2A5YQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.468846415 (381h55m19.020316906s) [noise=true , address=145.79.227.5:4222 , piece_id=JVGVDN76WQXEQPURRSDLGSFGRZYLM2O5JXCDBGYHDYOA6GWLGK6A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:20.490284295 (381h55m19.041754756s) [noise=true , address=145.79.227.39:4015 , piece_id=WWNLAWRDCGPSZLTADPJ2E72S6Z23VTLBO3K7ERUO5ZHVO7ZFHW3Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.336009608 (381h55m19.887480069s) [noise=true , address=145.79.227.20:4437 , piece_id=DQFITDE25PY77RFNFTUHJQNXPGYRSK63DUPJWNGCLZWXKOSVGFGA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:21.605937059 (381h55m20.15740752s) [noise=true , address=145.79.227.22:4511 , piece_id=AAURM3GVICXFDVAPM75MDV6S5SUUHKCFHEZRC6LP74YDSO3S376Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:22.462781333 (381h55m21.014251784s) [noise=true , address=145.79.227.32:4834 , piece_id=VOFW6VBFNYQ27CYZTILHU2G57AXWXVGZIXY6IJAGXLPBXQOHCQYQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:25.462743457 (381h55m24.014213918s) [noise=true , address=145.79.227.2:4115 , piece_id=SP5KFCEZKR467UHI4JQPFW32VTHEVNOYFVZXXLIHUU75DM6KSUTQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: <nil> Nov 25 05:59:26.107237118 (381h55m24.658707579s) [noise=true , address=145.79.227.28:4727 , piece_id=3KANZX4KMNJQ6SNEKATUN433WJASNR53WDR7Q3S24XHBQ5TNTFEA, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12LWaXPeH6XghWc64oeseaM52Tk2PQZWDDj1nVnYikHGz7PwEQ3, address:145.79.227.28:4727): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107296762 (381h55m24.658767223s) [noise=true , address=145.79.227.25:4617 , piece_id=EY3KCKCOQ4XV6TAJ3KRWYRZRKBKDDBUJY6ZGDALTYT3UQSXMANUQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12ifHtxXNY419Q7tJgZbBoz7ZULJmsob3QEgoGTZbhCPXH9TVQw, address:145.79.227.25:4617): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107510018 (381h55m24.658980479s) [noise=true , address=145.79.227.40:4050 , piece_id=WRTALIQ32JXE4MGSKQ2UT5ENV4MTRPMIA6HLHSNMADZMVUL27W7Q, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64, address:145.79.227.40:4050): protocol: expected piece hash; context deadline exceeded; EOF Nov 25 05:59:26.107623073 (381h55m24.659093534s) [noise=true , address=145.79.227.15:4875 , piece_id=O5HNO6E4HOKZBOFB2KHCB5CXZFU3XGEHHPJI43BUHMR5RZHRX36A, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: read tcp 188.166.225.170:47108->145.79.227.15:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107934426 (381h55m24.659404897s) [noise=true , address=145.79.227.30:4763 , piece_id=5GMTX7WYD733KWKNVEWN7DF7X66JSZPSJWEEUFD5VCLBTSJOGIJQ, seg_pos={0 0}, upload=336349]: Done uploading piece. err: ecclient: upload failed (node:12Xy6Y953cfdoyywnf98XTrbd9WuDZNEV8SRhkBKbZRHrU2ymPg, address:145.79.227.30:4763): protocol: read tcp 188.166.225.170:44844->145.79.227.30:4763: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 25 05:59:26.107973070 (381h55m24.659443531s) [seg_pos={0 0}, upload=336349]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Nov 25 05:59:26.520867304 (381h55m25.072337765s) [upload=336349]: Done uploading segments... Nov 25 05:59:26.520937578 (381h55m25.072408039s) [upload=336349]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 5.919499681s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [................................X...........................] duration: 3.366742229s trace: https://jaeger.dev.storj.io/trace/6e9e391a4d3f33cf FAIL (2025-11-25 06:09:32 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.366771637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.708409818s trace: https://jaeger.dev.storj.io/trace/3398168cd03d5a88 FAIL (2025-11-25 06:09:09 UTC) orbiter warning: operation: DownloadObject uplink: metaclient: context deadline exceeded Duration: 5.708437781s core uplink us1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.remote)