Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.463874082s FAIL (2025-06-29 04:06:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.463900483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.589503652s FAIL (2025-06-29 04:05:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.589530402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.589106762s FAIL (2025-06-29 04:05:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.589134625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.696460436s FAIL (2025-06-29 04:04:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.696488912s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.712356183s FAIL (2025-06-29 04:02:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.712383806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.710627217s FAIL (2025-06-29 04:02:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.710651294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.68549683s FAIL (2025-06-29 04:01:26 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.685524764s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.840441359s FAIL (2025-06-29 04:00:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.840463201s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.671617768s FAIL (2025-06-29 04:00:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.671712209s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.654647847s FAIL (2025-06-29 03:59:46 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.654671083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.688134623s FAIL (2025-06-29 03:58:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.6881641s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.624031985s FAIL (2025-06-29 03:57:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.624056702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.704911769s FAIL (2025-06-29 03:56:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.704938599s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.649745395s FAIL (2025-06-29 03:55:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.649772829s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.673922439s FAIL (2025-06-29 03:54:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.673949612s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.641407596s FAIL (2025-06-29 03:53:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.641434997s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.727548305s FAIL (2025-06-29 03:52:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.727574236s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.764194291s FAIL (2025-06-29 03:51:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.764217264s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.635927012s FAIL (2025-06-29 03:50:13 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.635953091s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.70968545s FAIL (2025-06-29 03:49:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.709704005s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.454988434s FAIL (2025-06-29 03:49:32 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.455018021s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.636557317s FAIL (2025-06-29 03:48:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.636582456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.586812958s FAIL (2025-06-29 03:48:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.586839449s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 5.626662841s FAIL (2025-06-29 03:47:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.626686157s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 5.61530101s FAIL (2025-06-29 03:46:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.615336127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 5.671737815s FAIL (2025-06-29 03:45:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.67177203s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 5.661815477s FAIL (2025-06-29 03:44:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.661845896s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 5.595453306s FAIL (2025-06-29 03:44:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.595480818s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 5.650627667s FAIL (2025-06-29 03:43:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.650657354s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 5.720717329s FAIL (2025-06-29 03:42:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.720742317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 5.630173127s FAIL (2025-06-29 03:41:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.630207292s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 5.692810149s FAIL (2025-06-29 03:40:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.692839747s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 5.646597569s FAIL (2025-06-29 03:40:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.646627625s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 5.687166484s FAIL (2025-06-29 03:39:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.68720166s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 5.658841654s FAIL (2025-06-29 03:39:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.658871129s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 5.703625393s FAIL (2025-06-29 03:38:16 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.703659308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.627532204s FAIL (2025-06-29 03:37:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.627556751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 5.767148359s FAIL (2025-06-29 03:36:54 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.767176833s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.622816234s FAIL (2025-06-29 03:35:41 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.622844278s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 5.563395184s FAIL (2025-06-29 03:34:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.563428208s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 5.612899091s FAIL (2025-06-29 03:33:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.612965097s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.231698186s FAIL (2025-06-29 03:33:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ptimal: 13 errs: <nil> Jun 29 03:31:53.992618646 (138h21m28.763259981s) [upload=110558]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jun 29 03:31:53.992623886 (138h21m28.763265221s) [upload=110558]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jun 29 03:31:54.284045604 (138h21m29.054686939s) [upload=110558]: Done uploading segments... Jun 29 03:31:54.284065682 (138h21m29.054707017s) [upload=110558]: Upload finished. err: <nil> Jun 29 03:32:53.064412972 (138h22m27.835054307s) [upload=110573]: Starting upload Jun 29 03:32:53.064473317 (138h22m27.835114652s) [upload=110573]: Done starting upload Jun 29 03:32:53.064639555 (138h22m27.83528089s) [upload=110573]: Uploading segments... Jun 29 03:32:53.064669713 (138h22m27.835311058s) [upload=110573]: Got next segment. Inline: false Jun 29 03:32:53.064692106 (138h22m27.835333441s) [upload=110573]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jun 29 03:32:53.064696384 (138h22m27.835337719s) [upload=110573]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jun 29 03:32:53.296363313 (138h22m28.067004648s) [seg_pos={0 0}, upload=110573]: Begin upload segment... Jun 29 03:32:53.296403770 (138h22m28.067045105s) [seg_pos={0 0}, upload=110573]: Done begin upload segment. Jun 29 03:32:53.296423738 (138h22m28.067065073s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296473103 (138h22m28.067114438s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296789648 (138h22m28.067430973s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.296896923 (138h22m28.067538258s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297061198 (138h22m28.067702533s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297227616 (138h22m28.067868991s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297320243 (138h22m28.067961578s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297520557 (138h22m28.068161882s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297564410 (138h22m28.068205745s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297775705 (138h22m28.06841703s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.297833014 (138h22m28.068474349s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298078154 (138h22m28.068719489s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298290680 (138h22m28.068932015s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298489350 (138h22m28.069130685s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298686006 (138h22m28.069327341s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:53.298873365 (138h22m28.0695147s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Uploading piece... Jun 29 03:32:54.970829162 (138h22m29.741470497s) [upload=110573]: Next returned nil segment Jun 29 03:32:54.970858468 (138h22m29.741499803s) [upload=110573]: Waiting for error group managing segments... Jun 29 03:32:55.253086457 (138h22m30.023727793s) [noise=true , address=104.36.82.13:4186 , piece_id=V6GKNUFGJPLY6RUEOMCLGHR5R3E4AE2ANY7K2N34ZEYAGBCBCAOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.263686566 (138h22m30.034327891s) [noise=true , address=104.36.82.3:4767 , piece_id=V2FX6WIKDGJYCOW3N7LJHP7OHNNIY6TEEEHG5WVFDKUGIT4UWVIQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.280783876 (138h22m30.051425211s) [noise=true , address=104.36.82.16:4294 , piece_id=N63CURTFLVCOM4GDRJCRCGSQZ2CL4XXO5RKCOF2KNPHVOL2OX6LQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.292731273 (138h22m30.063372608s) [noise=true , address=104.36.82.15:4259 , piece_id=26DXVVCKQ4TEI6LWVLW44XKQH6TDJVQSADKEDSEKQFKZU34VPLHQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.298803200 (138h22m30.069444535s) [noise=true , address=104.36.82.13:4185 , piece_id=JLJYFCLQNVQSKUUNIABMQNAV2GDU2VCECRDQRTM5VXMA4L6S2XTQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.321344183 (138h22m30.091985518s) [noise=true , address=104.36.82.8:4969 , piece_id=CPF6NN3VPXO6PO5SR77YCRD7S3UWWK66G2GBCH2GJ5E7GQQQ6CSA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.446686720 (138h22m30.217328055s) [noise=true , address=45.91.116.5:5298 , piece_id=CCWOAWQQFI3SGR3HU6NIGNOQMCYJE7PQUSZAWN7SETK254TJZV2A, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.517955756 (138h22m30.288597091s) [noise=true , address=45.91.116.8:4048 , piece_id=HZVZIMR34T4Y4UL2DZYTLF5Y5ENMGXGXV6JLED5HFXC4T7VEHJOQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.525841823 (138h22m30.296483158s) [noise=true , address=45.91.116.11:4117 , piece_id=EJDFZJ2H7MC6S26SO5LNO6JTJB4F2W6PN74WHK3G2C6B3RQOZNHA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.568066586 (138h22m30.338707911s) [noise=true , address=104.36.82.4:4801 , piece_id=GQWIUCRYXR5GWJSU44MW2WCMTAG5LHYUJXZOAWEJC2LAJQK6NPEQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:55.570973417 (138h22m30.341614752s) [noise=true , address=45.91.116.14:4190 , piece_id=YHVKC4WJUYLBY3FJEAZSVCPT4BKRGNYAECK2NQ2YDX36NMZYQYWA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:32:56.715200119 (138h22m31.485841454s) [noise=true , address=104.36.82.2:4729 , piece_id=VUS4VKFKINU3KRY3A4K2AWDB7JAAW4IBF6CVI7LCSLBMZM2LFSQQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: <nil> Jun 29 03:33:22.415980535 (138h22m57.18662187s) [noise=true , address=77.90.31.17:4772 , piece_id=VRE62K7YICWJ2UWFTQYPEJWZBTBRWISAD57USOD7M3CZ34K5WXSQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:18CNfMYrF1Gqgabi62fG23CagDZHW2QyZZNYKKTSjWQKLS4ppE, address:77.90.31.17:4772): protocol: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49434->77.90.31.17:4772: use of closed network connection Jun 29 03:33:22.416574742 (138h22m57.187216067s) [noise=true , address=77.90.31.13:4627 , piece_id=IZ5HJ325T33TKJLJRM5EHNSTZCSAMSVIKSSMISDAXZVQXKTJ2VQA, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, address:77.90.31.13:4627): protocol: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42554->77.90.31.13:4627: use of closed network connection Jun 29 03:33:22.416639305 (138h22m57.18728064s) [noise=true , address=77.90.31.21:4877 , piece_id=GQT7ZOBHSHGNRKFUM2SQTMQLTMFENTO65CXQHWWFOZUYT6ZFLE5Q, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, address:77.90.31.21:4877): protocol: read tcp 188.166.225.170:54178->77.90.31.21:4877: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416646709 (138h22m57.187288044s) [noise=true , address=77.90.31.12:4592 , piece_id=TGXQRWJ3JK2AKVALBON5AEROWQUOEKZB3NATJVJMQAVEERO7QHJQ, seg_pos={0 0}, upload=110573]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: read tcp 188.166.225.170:39612->77.90.31.12:4592: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jun 29 03:33:22.416673000 (138h22m57.187314335s) [seg_pos={0 0}, upload=110573]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jun 29 03:33:22.646425525 (138h22m57.41706687s) [upload=110573]: Done uploading segments... Jun 29 03:33:22.646449652 (138h22m57.417090987s) [upload=110573]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.231712233s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.39487102s FAIL (2025-06-28 13:03:57 UTC) orbiter warning: context deadline exceeded Duration: 6.394890106s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.833407208s FAIL (2025-06-28 13:03:28 UTC) orbiter warning: context deadline exceeded Duration: 5.833430853s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.096619417s FAIL (2025-06-28 13:03:08 UTC) orbiter warning: context deadline exceeded Duration: 6.096645449s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.109451042s FAIL (2025-06-28 13:02:25 UTC) orbiter warning: context deadline exceeded Duration: 6.109470889s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.96977134s FAIL (2025-06-28 13:01:10 UTC) orbiter warning: context deadline exceeded Duration: 5.969795926s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.467981813s FAIL (2025-06-28 13:00:25 UTC) orbiter warning: context deadline exceeded Duration: 6.468045776s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.938793274s FAIL (2025-06-28 12:59:48 UTC) orbiter warning: context deadline exceeded Duration: 5.938813071s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 6.241398777s FAIL (2025-06-28 12:59:22 UTC) orbiter warning: context deadline exceeded Duration: 6.241423525s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)