Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.710144512s FAIL (2025-07-16 13:28:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.710172365s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 10 out of 60 is failed (threshold=3) [................................................XX..XXXXXXXX] duration: 30.001072686s FAIL (2025-07-16 13:28:00 UTC) uplink: metainfo: metaclient: context deadline exceeded Duration: 30.001082844s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................X..X..XXXX] duration: 34.052920348s FAIL (2025-07-16 13:27:52 UTC) uplink: metaclient: context deadline exceeded Duration: 34.052932993s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XX..XXXXXXX] duration: 30.001273921s FAIL (2025-07-16 13:27:30 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001295692s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 5 out of 60 is failed (threshold=3) [...................................................X..X..XXX] duration: 37.304062264s FAIL (2025-07-16 13:27:18 UTC) uplink: metaclient: context deadline exceeded Duration: 37.30407585s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.828904098s FAIL (2025-07-16 13:27:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.828930327s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 8 out of 60 is failed (threshold=3) [..................................................XX..XXXXXX] duration: 30.00085427s FAIL (2025-07-16 13:26:45 UTC) uplink: metaclient: context deadline exceeded Duration: 30.000873405s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 4 out of 60 is failed (threshold=3) [....................................................X..X..XX] duration: 37.472651984s FAIL (2025-07-16 13:26:40 UTC) uplink: metaclient: context deadline exceeded Duration: 37.47267651s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 7 out of 60 is failed (threshold=3) [...................................................XX..XXXXX] duration: 30.001436291s FAIL (2025-07-16 13:26:01 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001454837s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.749526419s FAIL (2025-07-16 13:25:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.749553792s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.749568865s FAIL (2025-07-16 13:25:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.749600055s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.....................................................X..X..X] duration: 30.004044584s FAIL (2025-07-16 13:25:20 UTC) uplink: metaclient: context deadline exceeded Duration: 30.004065795s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 6 out of 60 is failed (threshold=3) [....................................................XX..XXXX] duration: 30.000613362s FAIL (2025-07-16 13:24:58 UTC) uplink: metaclient: context deadline exceeded Duration: 30.000624994s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [......................................................X..X..] duration: 30.002492481s FAIL (2025-07-16 13:24:47 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 30.00251296s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.854896735s FAIL (2025-07-16 13:24:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.854926613s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 5 out of 60 is failed (threshold=3) [.....................................................XX..XXX] duration: 30.000844146s FAIL (2025-07-16 13:24:28 UTC) uplink: metaclient: context deadline exceeded Duration: 30.000857671s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 4 out of 60 is failed (threshold=3) [......................................................XX..XX] duration: 30.001476968s FAIL (2025-07-16 13:23:57 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001499452s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......................................................X..X.] duration: 23.046852699s FAIL (2025-07-16 13:23:50 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 23.046876314s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.665264291s FAIL (2025-07-16 13:23:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.665288157s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration: 30.001336011s FAIL (2025-07-16 13:23:19 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001360158s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.883206659s FAIL (2025-07-16 13:23:06 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.883234512s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................X..X] duration: 44.771675853s FAIL (2025-07-16 13:22:58 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 44.771694799s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.681828355s FAIL (2025-07-16 13:21:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.681851859s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 8.503331581s FAIL (2025-07-16 13:21:49 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 8.503355306s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 709.387289ms FAIL (2025-07-16 13:21:46 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 709.413741ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 16.457531418s FAIL (2025-07-16 13:21:23 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 16.457602053s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 14.758509043s FAIL (2025-07-16 13:21:22 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 14.758534282s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 35.324031111s FAIL (2025-07-16 13:20:57 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 35.324054024s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.889701405s FAIL (2025-07-16 13:20:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.889728218s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 30.00047067s FAIL (2025-07-16 13:20:40 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 30.000494608s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000633198s FAIL (2025-07-16 13:19:53 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 30.000653898s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.750510546s FAIL (2025-07-16 13:19:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.750538499s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.659088228s FAIL (2025-07-16 13:19:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.659108376s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.889212512s FAIL (2025-07-16 13:18:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.889240483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.876699536s FAIL (2025-07-16 13:17:50 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.876724394s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.92322924s FAIL (2025-07-16 13:17:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.923254769s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.712283429s FAIL (2025-07-16 13:16:43 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.712307325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.763082737s FAIL (2025-07-16 13:16:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.763113646s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 5.816217431s FAIL (2025-07-16 13:15:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.816244402s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.115683835s FAIL (2025-07-16 13:14:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.115717469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.76443906s FAIL (2025-07-16 13:13:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.764460522s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.964395624s FAIL (2025-07-16 13:12:45 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.964422434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.706102544s FAIL (2025-07-16 13:12:17 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.706123965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.81983949s FAIL (2025-07-16 13:11:21 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.819866162s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.77502196s FAIL (2025-07-16 13:10:52 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.775053308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.768317419s FAIL (2025-07-16 13:10:29 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.768354161s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.748028254s FAIL (2025-07-16 13:09:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.748057469s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 5.716156873s FAIL (2025-07-16 13:08:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.716181911s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.853654231s FAIL (2025-07-16 13:08:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.853678387s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.711477961s FAIL (2025-07-16 13:07:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.711502448s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)