Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.899804731s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:12:06 UTC) orbiter warning: context deadline exceeded Duration: 5.899837686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 20.121226187s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:12:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 20.121273117s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 8 out of 60 is failed (threshold=6) [...........X.X....X.........X...X.XX.X......................] duration: 3.437032847s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:11:53 UTC) orbiter warning: context deadline exceeded Duration: 3.437056222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 6.72123746s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:11:36 UTC) orbiter warning: context deadline exceeded Duration: 6.721267076s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [X...........X.X....X.........X...X.XX.X.....................] duration: 3.447136696s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:11:28 UTC) orbiter warning: context deadline exceeded Duration: 3.447163747s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.690131746s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:11:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.690161684s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 9.565223184s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:10:57 UTC) orbiter warning: context deadline exceeded Duration: 9.565324989s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.X...........X.X....X.........X...X.XX.X....................] duration: 3.584569634s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:10:40 UTC) orbiter warning: context deadline exceeded Duration: 3.584597566s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [..X...........X.X....X.........X...X.XX.X...................] duration: 3.509623622s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:10:16 UTC) orbiter warning: context deadline exceeded Duration: 3.509684448s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 13.63974989s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:10:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.639777635s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 13.053473264s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:10:02 UTC) orbiter warning: context deadline exceeded Duration: 13.053594566s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 14.279416015s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:09:32 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.279444149s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 12.056012911s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:09:18 UTC) orbiter warning: context deadline exceeded Duration: 12.056120315s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [...X...........X.X....X.........X...X.XX.X..................] duration: 4.355759432s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:09:10 UTC) orbiter warning: context deadline exceeded Duration: 4.355794609s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 9.980881581s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:09:01 UTC) orbiter warning: context deadline exceeded Duration: 9.980912028s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 14.954560298s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:09:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.954590276s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 13.134913714s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:08:38 UTC) orbiter warning: context deadline exceeded Duration: 13.134944713s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 13.555898782s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:08:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.555930983s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [....X...........X.X....X.........X...X.XX.X.................] duration: 3.672969432s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:08:06 UTC) orbiter warning: context deadline exceeded Duration: 3.673038204s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 10.415930387s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:07:52 UTC) orbiter warning: context deadline exceeded Duration: 10.41595918s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 11.370130599s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:07:34 UTC) orbiter warning: context deadline exceeded Duration: 11.370176686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 13.607496372s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:07:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.607525449s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.....X...........X.X....X.........X...X.XX.X................] duration: 3.415931669s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:07:22 UTC) orbiter warning: context deadline exceeded Duration: 3.416001522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 13.462474025s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:06:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.462503642s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.634958096s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:06:20 UTC) orbiter warning: context deadline exceeded Duration: 6.634988754s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [......X...........X.X....X.........X...X.XX.X...............] duration: 3.559734585s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:06:20 UTC) orbiter warning: context deadline exceeded Duration: 3.559764472s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 14.05061564s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:05:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.050642422s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.001302025s trace: https://jaeger.dev.storj.io/trace/3e74c93562038233 FAIL (2025-11-22 17:05:33 UTC) orbiter warning: context deadline exceeded Duration: 30.001344585s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 9 out of 60 is failed (threshold=6) [.......X...........X.X....X.........X...X.XX.X..............] duration: 4.94099797s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:05:24 UTC) orbiter warning: context deadline exceeded Duration: 4.941033189s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 15.329530056s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:05:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.329564251s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [X.......X...........X.X....X.........X...X.XX.X.............] duration: 3.456534192s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:04:18 UTC) orbiter warning: context deadline exceeded Duration: 3.456562757s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 14.15472411s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:04:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.154759018s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [.X.......X...........X.X....X.........X...X.XX.X............] duration: 3.561955934s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:03:46 UTC) orbiter warning: context deadline exceeded Duration: 3.562055875s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 13.469081719s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:03:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.469107828s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [..X.......X...........X.X....X.........X...X.XX.X...........] duration: 3.594010522s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:02:56 UTC) orbiter warning: context deadline exceeded Duration: 3.594043044s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 13.83004286s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:02:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.830074471s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 10 out of 60 is failed (threshold=6) [...X.......X...........X.X....X.........X...X.XX.X..........] duration: 3.593998911s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:01:49 UTC) orbiter warning: context deadline exceeded Duration: 3.594110032s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 13.714428141s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:01:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 13.714477655s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [X...X.......X...........X.X....X.........X...X.XX.X.........] duration: 5.001126205s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:01:17 UTC) orbiter warning: context deadline exceeded Duration: 5.001176892s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.388486048s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:00:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 15.388518299s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X....................................X......................] duration: 14.546362988s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 17:00:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 14.546393518s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 11 out of 60 is failed (threshold=6) [.X...X.......X...........X.X....X.........X...X.XX.X........] duration: 3.592009613s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 17:00:05 UTC) orbiter warning: context deadline exceeded Duration: 3.592047564s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X.X...X.......X...........X.X....X.........X...X.XX.X.......] duration: 3.537578757s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 16:59:29 UTC) orbiter warning: context deadline exceeded Duration: 3.537605668s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX....................................X.....................] duration: 12.525774627s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 16:59:20 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 12.525805815s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X.X...X.......X...........X.X....X.........X...X.XX.X......] duration: 3.443257152s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 16:59:01 UTC) orbiter warning: context deadline exceeded Duration: 3.443286698s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.XX....................................X....................] duration: 11.689213389s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 16:58:14 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.689241262s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X.X...X.......X...........X.X....X.........X...X.XX.X.....] duration: 3.445825081s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 16:57:47 UTC) orbiter warning: context deadline exceeded Duration: 3.445893823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..XX....................................X...................] duration: 10.353008732s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 16:57:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 10.353031765s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X.X...X.......X...........X.X....X.........X...X.XX.X....] duration: 4.405222151s trace: https://jaeger.dev.storj.io/trace/1c5d4c55f354b821 FAIL (2025-11-22 16:57:08 UTC) orbiter warning: context deadline exceeded Duration: 4.405359471s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...XX....................................X..................] duration: 11.974411816s trace: https://jaeger.dev.storj.io/trace/73373c215464be3b FAIL (2025-11-22 16:56:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Nov 22 16:41:49.669209413 (320h37m48.220679874s) [upload=282381]: Starting upload Nov 22 16:41:49.669269467 (320h37m48.220739928s) [upload=282381]: Done starting upload Nov 22 16:41:49.669428691 (320h37m48.220899152s) [upload=282381]: Uploading segments... Nov 22 16:41:49.669491310 (320h37m48.220961761s) [upload=282381]: Got next segment. Inline: false Nov 22 16:41:49.669533520 (320h37m48.221003971s) [upload=282381]: Flush batch item: pb.BatchRequestItem_ObjectBegin Nov 22 16:41:49.669543579 (320h37m48.22101405s) [upload=282381]: Flush batch item: pb.BatchRequestItem_SegmentBegin Nov 22 16:41:50.245063880 (320h37m48.796534341s) [seg_pos={0 0}, upload=282381]: Begin upload segment... Nov 22 16:41:50.245111712 (320h37m48.796582173s) [seg_pos={0 0}, upload=282381]: Done begin upload segment. Nov 22 16:41:50.245193718 (320h37m48.796664179s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245216191 (320h37m48.796686652s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.245864785 (320h37m48.797335256s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246394764 (320h37m48.797865235s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.246736996 (320h37m48.798207447s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247026979 (320h37m48.79849743s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247332229 (320h37m48.798802691s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247652190 (320h37m48.799122641s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.247934046 (320h37m48.799404497s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248316444 (320h37m48.799786905s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.248789244 (320h37m48.800259695s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249041134 (320h37m48.800511585s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249170470 (320h37m48.800640921s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249465162 (320h37m48.800935613s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.249550193 (320h37m48.801020654s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:50.250091574 (320h37m48.801562026s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Uploading piece... Nov 22 16:41:51.914100898 (320h37m50.465571359s) [upload=282381]: Next returned nil segment Nov 22 16:41:51.914115246 (320h37m50.465585707s) [upload=282381]: Waiting for error group managing segments... Nov 22 16:41:52.302424073 (320h37m50.853894534s) [noise=true , address=45.91.116.7:4009 , piece_id=VSTM2QNABJSQRYGB763T757ARHMCF2BIFI4RLNZ33HY65B264USA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.323996711 (320h37m50.875467172s) [noise=true , address=45.91.116.13:4154 , piece_id=PJ4ABV44OHMZDGOX7UQFUK3ENI5JTXLJ2LNKZO7A7RQPJD53WSFQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.366609934 (320h37m50.918080395s) [noise=true , address=77.90.31.13:4627 , piece_id=VPMNLZJDHW27RXRUMVKA7HPLF6TXKAQLXCKQ2GDWVA2HWCE72WIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.383661573 (320h37m50.935132035s) [noise=true , address=77.90.31.10:4517 , piece_id=FZZW7RIQZE6P5J6OKUFCLK5X6SI67QGRHCPJVZPSJDQ6A5VBLFBA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:52.921072257 (320h37m51.472542718s) [noise=true , address=77.90.31.12:4592 , piece_id=X2D67ERUFKBRZR3ND74GB7OMZ2LGWWKYDWR63PTGH4YMNE5E6MDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.072975457 (320h37m51.624445918s) [noise=true , address=77.90.31.21:4878 , piece_id=3H2XE3HXTKMKKVZXFFTSKCB3IM4X74OXLAX7TJOF6ISDGSEO4REQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:53.919460602 (320h37m52.470931063s) [noise=true , address=45.91.116.9:4262 , piece_id=LERQ3223LFXZGWMNPXN5KANRIIGHURV2V2GSC6QTWYNSCQG37OJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:41:59.286028452 (320h37m57.837498913s) [noise=true , address=145.79.227.15:4875 , piece_id=LU2QSMICPBLNAKC7JZNAD7XBWPCVWPMPX3KIKX2V7P44X2XGXJLA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:00.186876102 (320h37m58.738346563s) [noise=true , address=145.79.227.21:4474 , piece_id=FE2KMZIEQSYGCHRM7LCRMFDVOP35RSLTEW2LC3DVGCYT5HVEKPIA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:02.278437229 (320h38m0.82990769s) [noise=true , address=145.79.227.30:4761 , piece_id=GF525VXMQ25FPBILIKRRQ6X35HSSF73RICFKPN3ODS6IXASCQVJQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:04.727839046 (320h38m3.279309507s) [noise=true , address=145.79.227.2:4113 , piece_id=QEPFSWYRVU2QI6N63UIGAEO6XNGAY3N4LFGK23QDBMQHO5WXYJAA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:05.006455109 (320h38m3.557925571s) [noise=true , address=145.79.227.14:4837 , piece_id=YUEXSNF3D6BTVVTTDLIYU53TMY44V5YJ6ARVGHLVNT6FPQ4JCXDA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: <nil> Nov 22 16:42:18.025379977 (320h38m16.576850438s) [noise=true , address=145.79.227.23:4545 , piece_id=NNWS5ZTGLI7NFQT2AULIK6CUUEUSFYR3GSEGFZDXDGHFOGH2OTZQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12P5cnEza1Xxwg4qo1nau11zYHsHxqEAwhFsnt2UE1RhtTrJmra, address:145.79.227.23:4545): protocol: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40154->145.79.227.23:4545: use of closed network connection Nov 22 16:42:18.026301482 (320h38m16.577771934s) [noise=true , address=145.79.227.31:4799 , piece_id=AYXSVOS7ONCJ4W4VKMHYJ2YAJ54QYBGYFSUF2A4LANKOII5HQ7EA, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, address:145.79.227.31:4799): protocol: read tcp 188.166.225.170:42764->145.79.227.31:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026393338 (320h38m16.577863789s) [noise=true , address=145.79.227.27:4690 , piece_id=3QNAIMKSFRF62T5OTZWOFSFXA5N6ZASLOUJVCC2UYIJV6GILITDQ, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12fyCro9gG2d3myk6Cju5vmf4CbkqbJN2PKAP8EbpcT1Rtao27H, address:145.79.227.27:4690): protocol: read tcp 188.166.225.170:43594->145.79.227.27:4690: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026594139 (320h38m16.57806461s) [noise=true , address=145.79.227.9:4365 , piece_id=SO5POI5NR77NPOPQYXPIYKC5ETT7HTWGVAP4NLBUEMBISZONBV4Q, seg_pos={0 0}, upload=282381]: Done uploading piece. err: ecclient: upload failed (node:12rvMymD9ejLTfLTfeF4VAycp2SdGUMBtR3o45x8JeMkCDWpCHq, address:145.79.227.9:4365): protocol: read tcp 188.166.225.170:56822->145.79.227.9:4365: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Nov 22 16:42:18.026628976 (320h38m16.578099437s) [seg_pos={0 0}, upload=282381]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Nov 22 16:42:18.509068735 (320h38m17.060539196s) [upload=282381]: Done uploading segments... Nov 22 16:42:18.509116086 (320h38m17.060586547s) [upload=282381]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 11.974442726s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)