Orbiter History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..............................XX..XXXXXXXXXX................] duration: 205.971911ms FAIL (2025-07-16 13:40:31 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 205.994826ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.................................X..X..XXXXXXXXX............] duration: 2.839413436s FAIL (2025-07-16 13:39:43 UTC) uplink: metaclient: context deadline exceeded Duration: 2.839439897s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...............................XX..XXXXXXXXXX...............] duration: 260.099175ms FAIL (2025-07-16 13:39:34 UTC) orbiter warning: uplink: metaclient: context deadline exceeded Duration: 260.126006ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [................................XX..XXXXXXXXXX..............] duration: 200.409147ms FAIL (2025-07-16 13:38:58 UTC) uplink: metaclient: context deadline exceeded Duration: 200.437212ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [..................................X..X..XXXXXXXXX...........] duration: 2.980898804s FAIL (2025-07-16 13:38:51 UTC) uplink: metaclient: context deadline exceeded Duration: 2.980925925s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.................................XX..XXXXXXXXXX.............] duration: 256.064295ms FAIL (2025-07-16 13:38:26 UTC) uplink: metaclient: context deadline exceeded Duration: 256.097688ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [...................................X..X..XXXXXXXXX..........] duration: 2.723683266s FAIL (2025-07-16 13:38:25 UTC) uplink: metaclient: context deadline exceeded Duration: 2.723715437s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..................................XX..XXXXXXXXXX............] duration: 194.93196ms FAIL (2025-07-16 13:38:09 UTC) uplink: metaclient: context deadline exceeded Duration: 194.961074ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [....................................X..X..XXXXXXXXX.........] duration: 2.833674816s FAIL (2025-07-16 13:37:50 UTC) uplink: metaclient: context deadline exceeded Duration: 2.833705735s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...................................XX..XXXXXXXXXX...........] duration: 205.302468ms FAIL (2025-07-16 13:37:42 UTC) uplink: metaclient: context deadline exceeded Duration: 205.327176ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.....................................X..X..XXXXXXXXX........] duration: 3.315845169s FAIL (2025-07-16 13:37:19 UTC) uplink: metaclient: context deadline exceeded Duration: 3.315876379s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.683521782s FAIL (2025-07-16 13:37:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.683544284s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [......................................X..X..XXXXXXXXX.......] duration: 4.084459347s FAIL (2025-07-16 13:37:02 UTC) uplink: metaclient: context deadline exceeded Duration: 4.084484895s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [....................................XX..XXXXXXXXXX..........] duration: 1.630842295s FAIL (2025-07-16 13:37:00 UTC) uplink: metaclient: context deadline exceeded Duration: 1.630861511s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.......................................X..X..XXXXXXXXX......] duration: 7.994393458s FAIL (2025-07-16 13:36:17 UTC) uplink: metaclient: context deadline exceeded Duration: 7.994414317s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.788225236s FAIL (2025-07-16 13:36:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.788255183s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.....................................XX..XXXXXXXXXX.........] duration: 5.592981113s FAIL (2025-07-16 13:35:55 UTC) uplink: metaclient: context deadline exceeded Duration: 5.593007695s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.800148347s FAIL (2025-07-16 13:35:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.800178754s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [......................................XX..XXXXXXXXXX........] duration: 269.047357ms FAIL (2025-07-16 13:35:22 UTC) uplink: metaclient: context deadline exceeded Duration: 269.072153ms core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.788495423s FAIL (2025-07-16 13:35:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.788519127s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [........................................X..X..XXXXXXXXX.....] duration: 3.38067725s FAIL (2025-07-16 13:35:02 UTC) uplink: metaclient: context deadline exceeded Duration: 3.380704482s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.......................................XX..XXXXXXXXXX.......] duration: 241.80908ms FAIL (2025-07-16 13:35:01 UTC) uplink: metaclient: context deadline exceeded Duration: 241.837886ms core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.........................................X..X..XXXXXXXXX....] duration: 4.187339389s FAIL (2025-07-16 13:34:37 UTC) uplink: metaclient: context deadline exceeded Duration: 4.187365178s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [........................................XX..XXXXXXXXXX......] duration: 3.46181984s FAIL (2025-07-16 13:34:08 UTC) uplink: metaclient: context deadline exceeded Duration: 3.461844718s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.729282223s FAIL (2025-07-16 13:33:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.729308985s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [..........................................X..X..XXXXXXXXX...] duration: 12.69565753s FAIL (2025-07-16 13:33:43 UTC) uplink: metaclient: context deadline exceeded Duration: 12.695681638s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [...........................................X..X..XXXXXXXXX..] duration: 16.303144557s FAIL (2025-07-16 13:33:25 UTC) uplink: metaclient: context deadline exceeded Duration: 16.30317208s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.........................................XX..XXXXXXXXXX.....] duration: 4.779000379s FAIL (2025-07-16 13:32:59 UTC) uplink: metaclient: context deadline exceeded Duration: 4.77903201s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.753131229s FAIL (2025-07-16 13:32:57 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.753157029s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [............................................X..X..XXXXXXXXX.] duration: 11.846066725s FAIL (2025-07-16 13:32:56 UTC) uplink: metaclient: context deadline exceeded Duration: 11.846092955s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..........................................XX..XXXXXXXXXX....] duration: 26.601125988s FAIL (2025-07-16 13:32:48 UTC) uplink: metaclient: context deadline exceeded Duration: 26.60115303s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [.............................................X..X..XXXXXXXXX] duration: 30.003883797s FAIL (2025-07-16 13:32:42 UTC) uplink: metaclient: context deadline exceeded Duration: 30.00390662s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.813153354s FAIL (2025-07-16 13:32:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.813182379s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [...........................................XX..XXXXXXXXXX...] duration: 27.644907045s FAIL (2025-07-16 13:31:54 UTC) uplink: metaclient: context deadline exceeded Duration: 27.644925471s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 10 out of 60 is failed (threshold=3) [..............................................X..X..XXXXXXXX] duration: 30.004886204s FAIL (2025-07-16 13:31:53 UTC) uplink: context deadline exceeded Duration: 30.004912775s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [............................................XX..XXXXXXXXXX..] duration: 27.165811087s FAIL (2025-07-16 13:31:26 UTC) uplink: metaclient: context deadline exceeded Duration: 27.165829822s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.775568162s FAIL (2025-07-16 13:31:03 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.775594473s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [.............................................XX..XXXXXXXXXX.] duration: 29.57683367s FAIL (2025-07-16 13:30:58 UTC) uplink: metaclient: context deadline exceeded Duration: 29.576853218s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 9 out of 60 is failed (threshold=3) [...............................................X..X..XXXXXXX] duration: 30.003706589s FAIL (2025-07-16 13:30:48 UTC) uplink: metaclient: context deadline exceeded Duration: 30.003725997s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.004330125s FAIL (2025-07-16 13:30:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 6.00435312s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.731064896s FAIL (2025-07-16 13:29:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.731089854s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 12 out of 60 is failed (threshold=3) [..............................................XX..XXXXXXXXXX] duration: 30.001398574s FAIL (2025-07-16 13:29:49 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001420136s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 8 out of 60 is failed (threshold=3) [................................................X..X..XXXXXX] duration: 30.003175792s FAIL (2025-07-16 13:29:34 UTC) uplink: metaclient: context deadline exceeded Duration: 30.003196141s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.762678548s FAIL (2025-07-16 13:29:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.762707252s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 7 out of 60 is failed (threshold=3) [.................................................X..X..XXXXX] duration: 30.003058739s FAIL (2025-07-16 13:28:38 UTC) uplink: metaclient: context deadline exceeded Duration: 30.003100469s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 11 out of 60 is failed (threshold=3) [...............................................XX..XXXXXXXXX] duration: 30.000768404s FAIL (2025-07-16 13:28:34 UTC) uplink: metaclient: context deadline exceeded Duration: 30.000791828s core uplink ap1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.710144512s FAIL (2025-07-16 13:28:11 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== upload Jul 16 12:49:54.454430745 (555h39m29.22507217s) [upload=444362]: Uploading segments... Jul 16 12:49:54.454612783 (555h39m29.225254118s) [upload=444362]: Got next segment. Inline: false Jul 16 12:49:54.454649724 (555h39m29.225291059s) [upload=444362]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 16 12:49:54.454655314 (555h39m29.225296649s) [upload=444362]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 16 12:49:54.699076943 (555h39m29.469718278s) [seg_pos={0 0}, upload=444362]: Begin upload segment... Jul 16 12:49:54.699118793 (555h39m29.469760118s) [seg_pos={0 0}, upload=444362]: Done begin upload segment. Jul 16 12:49:54.699140215 (555h39m29.46978155s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699150905 (555h39m29.46979224s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699200289 (555h39m29.469841614s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699491557 (555h39m29.470132892s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699533877 (555h39m29.470175203s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699737808 (555h39m29.470379143s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.699859721 (555h39m29.470501056s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700001562 (555h39m29.470642887s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700262442 (555h39m29.470903787s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700465520 (555h39m29.471106855s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700614956 (555h39m29.471256291s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.700835788 (555h39m29.471477113s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701059667 (555h39m29.471700992s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701294466 (555h39m29.471935811s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701455474 (555h39m29.472096809s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:54.701695604 (555h39m29.472336939s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Uploading piece... Jul 16 12:49:56.462804855 (555h39m31.23344618s) [upload=444362]: Next returned nil segment Jul 16 12:49:56.462827357 (555h39m31.233468692s) [upload=444362]: Waiting for error group managing segments... Jul 16 12:49:56.744618805 (555h39m31.51526014s) [noise=true , address=45.91.116.10:4083 , piece_id=TRF2LD6QTPGSJ6H3ZBO6ZA2BX3ZGUDK5PBZYXNGHUCHSTBDN5GTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.753612183 (555h39m31.524253508s) [noise=true , address=45.91.116.11:4119 , piece_id=YLYQYUSQQ3XFGQHATLKSMAKMEEZIJXSFGRHDNST3WPOWO7Z3D6QA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.761769189 (555h39m31.532410524s) [noise=true , address=45.91.116.9:4261 , piece_id=WZ7JMCI4O7FLMUZ3DFMOXW5SJBB24OCN6ZMKLLGJVFJZYCZULGVQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.771222316 (555h39m31.541863651s) [noise=true , address=45.91.116.12:4227 , piece_id=4VO2V6QRBIZK2MTURBDDY2NEJJ65MKR3JUINMZURAZNMBZMY6HNQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.791205801 (555h39m31.561847136s) [noise=true , address=45.91.116.8:4045 , piece_id=775P4F74ENG2MPEESXVP4E2YRUVIPAYFYJ7BRCKD6LFSDMWCFDTA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.834418742 (555h39m31.605060077s) [noise=true , address=45.91.116.10:4081 , piece_id=BLFVTGDFICTNBQLP5PDH3DWIS4JJYVKTD4Z4ADYWVOS62A23I2QQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838049959 (555h39m31.608691294s) [noise=true , address=45.91.116.13:4153 , piece_id=MVZ7IWOC76WEB7FLAEBYEXGGFKUF7ETEIE5NEVV6VXXYCBY5E2HA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.838929011 (555h39m31.609570346s) [noise=true , address=45.91.116.5:4297 , piece_id=YKATWUFEURM2TFYRRS32RYWORAIE6DQJDMSD2G5VJNVCR2TOANAQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.842586269 (555h39m31.613227604s) [noise=true , address=45.91.116.4:4940 , piece_id=RBCXABGY54Q6NUYS4HBJYUO735G5KEXFHMFDB7A43OLUMSTBVASA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:49:56.847874477 (555h39m31.618515812s) [noise=true , address=104.36.82.4:4802 , piece_id=DP3RGIYNJD7B2RGC75JDLDHSZPZFFWE2XINJ2OWEYLRJZITZKH3Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: <nil> Jul 16 12:50:23.796531007 (555h39m58.567172343s) [noise=true , address=77.90.31.18:4805 , piece_id=M72AUALYTYA5JSCFKBD6KQJMJMRCWVHNKLLOUA5HWFVD7ROXOIQQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796638163 (555h39m58.567279498s) [noise=true , address=77.90.31.13:4626 , piece_id=KPYO6FJN4CRMPASUY7QNVR7LGOEWJSOIN77QWYH6NQO5WEFJSUEQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34878->77.90.31.13:4626: use of closed network connection Jul 16 12:50:23.796656157 (555h39m58.567297492s) [noise=true , address=77.90.31.19:4843 , piece_id=S2PNAQ4O6TNV7J4D4MR7XLVYSN7K5MTR2R2NZHZWAVKFTZTQOJ6Q, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12akCPrMTihqq6CKv7VtDbHhXnWrXygXXhZKgZbftGbymRNzuMb, address:77.90.31.19:4843): protocol: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52290->77.90.31.19:4843: use of closed network connection Jul 16 12:50:23.796682628 (555h39m58.567323963s) [noise=true , address=77.90.31.11:4554 , piece_id=Z7BVO6PA7T56XUMN64H4RSSPTQLRTAMZXSKMDG2HMTYKQ2MR6IFA, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33128->77.90.31.11:4554: use of closed network connection Jul 16 12:50:23.796701113 (555h39m58.567342448s) [noise=true , address=77.90.31.14:4661 , piece_id=QJLO67GGOMAQKSMENVTF464Y65K6C3U5VTZ4R6QRTT26KV6G3TOQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, address:77.90.31.14:4661): protocol: expected piece hash; context deadline exceeded; EOF Jul 16 12:50:23.796730048 (555h39m58.567371383s) [noise=true , address=77.90.31.18:4807 , piece_id=75JF6AQRMDV6EBZYPMYAVU6OIJOQF6VWCU7GKHLAFOJLOTWT46MQ, seg_pos={0 0}, upload=444362]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50186->77.90.31.18:4807: use of closed network connection Jul 16 12:50:23.796753904 (555h39m58.567395239s) [seg_pos={0 0}, upload=444362]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 16 12:50:24.030501338 (555h39m58.801142683s) [upload=444362]: Done uploading segments... Jul 16 12:50:24.030531145 (555h39m58.80117248s) [upload=444362]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 5.710172365s core uplink us-select-1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 10 out of 60 is failed (threshold=3) [................................................XX..XXXXXXXX] duration: 30.001072686s FAIL (2025-07-16 13:28:00 UTC) uplink: metainfo: metaclient: context deadline exceeded Duration: 30.001082844s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 6 out of 60 is failed (threshold=3) [..................................................X..X..XXXX] duration: 34.052920348s FAIL (2025-07-16 13:27:52 UTC) uplink: metaclient: context deadline exceeded Duration: 34.052932993s core uplink ap1
checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline) antiblip: 9 out of 60 is failed (threshold=3) [.................................................XX..XXXXXXX] duration: 30.001273921s FAIL (2025-07-16 13:27:30 UTC) uplink: metaclient: context deadline exceeded Duration: 30.001295692s core uplink ap1

Orbiter Check History

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.ap1.inline)

checkers.Uplink{Satellite: "121RTSDpyNZVcEU84Ticf2L1ntiuUimbWgfATz21tuvgk3vzoA6@ap1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.ap1.inline)