Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.24476418s FAIL (2025-08-27 21:07:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.244796221s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 6.104890808s FAIL (2025-08-27 21:06:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 6.104916677s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...........X.....X.....XX....X......X...X...X...............] duration: 6.22538973s FAIL (2025-08-27 21:06:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.225416681s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 1.954967433s FAIL (2025-08-27 21:06:18 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.954996057s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [............X.....X.....XX....X......X...X...X..............] duration: 6.177863503s FAIL (2025-08-27 21:06:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.177880756s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.959724614s FAIL (2025-08-27 21:05:51 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.95974929s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........................X..........................] duration: 1.946088692s FAIL (2025-08-27 21:05:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.946117287s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.23678037s FAIL (2025-08-27 21:05:48 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.2368112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 5.934705976s FAIL (2025-08-27 21:05:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 5.934734691s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.............X.....X.....XX....X......X...X...X.............] duration: 6.170697781s FAIL (2025-08-27 21:05:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.170729011s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 6.258396408s FAIL (2025-08-27 21:05:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.25842317s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.297094339s FAIL (2025-08-27 21:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 6.297118735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.........................X.........................] duration: 2.088492627s FAIL (2025-08-27 21:05:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.088510522s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 1.959469766s FAIL (2025-08-27 21:04:57 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 1.959513309s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.33554925s FAIL (2025-08-27 21:04:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.335598424s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..............X.....X.....XX....X......X...X...X............] duration: 6.009270699s FAIL (2025-08-27 21:04:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.009296809s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........................X........................] duration: 1.934335854s FAIL (2025-08-27 21:04:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.93437577s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 5.846485898s FAIL (2025-08-27 21:04:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 5.846501456s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 2.039317447s FAIL (2025-08-27 21:04:21 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.039348566s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........................X.......................] duration: 1.94115403s FAIL (2025-08-27 21:03:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.941171974s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 2.058690314s FAIL (2025-08-27 21:03:45 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.05870909s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...............X.....X.....XX....X......X...X...X...........] duration: 6.342225082s FAIL (2025-08-27 21:03:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.3422495s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 6.228376645s FAIL (2025-08-27 21:03:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.228396663s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.029117382s FAIL (2025-08-27 21:03:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 6.029147s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........................X......................] duration: 1.925652827s FAIL (2025-08-27 21:03:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.925682084s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 2.212562993s FAIL (2025-08-27 21:02:56 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 2.21259183s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [................X.....X.....XX....X......X...X...X..........] duration: 5.987505071s FAIL (2025-08-27 21:02:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 5.98754153s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [............X.........................X.....................] duration: 2.028749937s FAIL (2025-08-27 21:02:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.028771327s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.019020835s FAIL (2025-08-27 21:02:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 6.019052635s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 16.300515147s FAIL (2025-08-27 21:02:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 16.300544905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.................X.....X.....XX....X......X...X...X.........] duration: 22.360782641s FAIL (2025-08-27 21:02:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 22.360801247s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 21.950209773s FAIL (2025-08-27 21:02:28 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 21.950225904s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 55.069095056s FAIL (2025-08-27 21:02:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled ======LOGS====== Aug 27 21:01:30.745282572 (3h23m9.90788351s) [upload=3529]: Starting upload Aug 27 21:01:30.745340883 (3h23m9.907941821s) [upload=3529]: Done starting upload Aug 27 21:01:30.745426136 (3h23m9.908027074s) [upload=3529]: Uploading segments... Aug 27 21:01:30.745508333 (3h23m9.908109302s) [upload=3529]: Got next segment. Inline: false Aug 27 21:01:30.745539824 (3h23m9.908140763s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 21:01:30.745544533 (3h23m9.908145471s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 21:01:31.280543219 (3h23m10.443144157s) [seg_pos={0 0}, upload=3529]: Begin upload segment... Aug 27 21:01:31.280587383 (3h23m10.443188321s) [seg_pos={0 0}, upload=3529]: Done begin upload segment. Aug 27 21:01:31.280612261 (3h23m10.443213199s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.280984203 (3h23m10.443585141s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281255883 (3h23m10.443856821s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281524486 (3h23m10.444125434s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.281782430 (3h23m10.444383368s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282007260 (3h23m10.444608198s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282310380 (3h23m10.444911318s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282567983 (3h23m10.445168921s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.282833631 (3h23m10.445434569s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283124839 (3h23m10.445725777s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283546204 (3h23m10.446147142s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.283891796 (3h23m10.446492734s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284106237 (3h23m10.446707175s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284359661 (3h23m10.446960599s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284644436 (3h23m10.447245374s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.284845451 (3h23m10.447446389s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285104877 (3h23m10.447705815s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:31.285331090 (3h23m10.447932028s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Uploading piece... Aug 27 21:01:32.410032973 (3h23m11.572633911s) [upload=3529]: Next returned nil segment Aug 27 21:01:32.410049694 (3h23m11.572650632s) [upload=3529]: Waiting for error group managing segments... Aug 27 21:01:32.703481276 (3h23m11.866082214s) [noise=true , address=79.127.205.234:4608 , piece_id=ER7ESGT5Z6BUHHPIKM36PC2KH7P7MA4PLYZCS2HGUHLDCIRTEUCQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.754433107 (3h23m11.917034045s) [noise=true , address=64.49.13.140:4509 , piece_id=4DAEPHJGP7ZHUEYKQQMEP23O6WYIPPET5ABXZKI2KXYD6EKXRWTQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:32.769316916 (3h23m11.931917844s) [noise=true , address=64.49.13.154:4934 , piece_id=KWJ3ESYHWGIQ5DAJTHTSV2CDT4DZSD5UC2RBWXM62ZBRUVWNL47Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.238239080 (3h23m12.400840018s) [noise=true , address=77.90.31.14:4696 , piece_id=SKSRHXRN3QFPW3MEQZVD7VBJOQPX5FBZI5HPFZDPGNR7A75FPTTA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.264951902 (3h23m12.427552841s) [noise=true , address=104.36.82.60:4337 , piece_id=JDPCW2QW2KNATPFSYWZW72LWAOVNTVU56AEJTDEUCBQ77WMAFX5A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.302245009 (3h23m12.464845947s) [noise=true , address=104.36.82.53:4136 , piece_id=2NP4QEFW7OOAL2BDFHQ22J2VIKV6T5HKDLGZUTI6VL5MMIELBYNA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.327002771 (3h23m12.489603709s) [noise=true , address=77.90.31.18:4819 , piece_id=LLCJQP2WEGPKLKWHYGX7DHOTQLCBK45IZQN6YHZ4E5M3E53Y54OA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.349616610 (3h23m12.512217548s) [noise=true , address=104.36.82.5:4862 , piece_id=UWXBMKYJRQQEGKSM5QSFMMFPYIXR35UDY3NRJ2W33NEUGWBYQEGQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.376179857 (3h23m12.538780795s) [noise=true , address=104.36.82.40:4951 , piece_id=U2G5KKDMBQQGG2WYEW4UI6FTOREW7GJO4WS5352RR5P3U7YT4CIQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.397364932 (3h23m12.55996587s) [noise=true , address=77.90.31.21:4889 , piece_id=RS4OJA4DMFFMBN462V5YM735QX22GNEICRBAS2SCRPIOBQME4D6Q, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.409545057 (3h23m12.572145985s) [noise=true , address=45.91.116.13:4160 , piece_id=S5ETLGYEMPWMFN2FCD6HXKAHU55OQ3TZX3PBEE4Y6HLKBC6ZX4UA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.437280667 (3h23m12.599881605s) [noise=true , address=45.91.116.9:4276 , piece_id=FRJTESATGFKGTHLIB76CZWAQCZ2ATGWXMV4XAHZHQXG4HYF3XHAA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.468508528 (3h23m12.631109466s) [noise=true , address=79.127.201.215:4347 , piece_id=YGGRZ2O35LZSK5KN7MQ7DVFXK4PBKEONOGV2RH5LCRM72GKIQJ4A, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475167368 (3h23m12.637768306s) [noise=true , address=104.36.82.8:4975 , piece_id=MCW76OZPJIHPJJKBPFU4BTCDB35YJVMYJTXSY24CMGGFNIWKQHLQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: <nil> Aug 27 21:01:33.475182628 (3h23m12.637783566s) [seg_pos={0 0}, upload=3529]: Segment reached optimal threshold of 14 pieces. Aug 27 21:01:33.475283131 (3h23m12.637884069s) [noise=true , address=77.90.31.10:4539 , piece_id=5WIVFFKFOZHZUAEHKOGPNYINLRDY5ZHB2KBXAA7MNV7A3K2O7GRA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DYHaVii53Q4FhtYNBMC64MP1uKHffSTMdfETK8kS94e4ZPfka): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475342244 (3h23m12.637943182s) [noise=true , address=104.36.82.54:4172 , piece_id=COW555UMDD7DIWMTEHKYJX3LTYOXQMI6LKV55JC7KG7BVNGJD7XQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1huxbetPkHjAivbfPrK3St4USc2KxPBBULxE1w7N6zC8rsoZMd): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475395966 (3h23m12.637996905s) [noise=true , address=77.90.31.12:4616 , piece_id=MFRA2SXJXNJBJE7BDDL5Y6S4BA5RINA7QHKYUVZ33YD6FKAY5JXQ, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qb6sMhCyzYd4S15Td1kzD7kgqNKSi2avG15CQRSG8bcDVvGzi): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475416385 (3h23m12.638017313s) [noise=true , address=45.91.116.11:4131 , piece_id=63FGKJCTE65WJCDFPKK4IFJLJSYOUQC4XPOOP6PZZJW624WNV3XA, seg_pos={0 0}, upload=3529]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124sCe84Td2UTjNKjgjFhaHbkkcLgvZY3Hn2LnAYYrsx5c14rUE): protocol: expected piece hash; context canceled; EOF Aug 27 21:01:33.475458656 (3h23m12.638059594s) [seg_pos={0 0}, upload=3529]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 21:01:33.475548838 (3h23m12.638149766s) [upload=3529]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 21:01:33.475553507 (3h23m12.638154435s) [upload=3529]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 21:02:25.127939378 (3h24m4.290540327s) [upload=3529]: Done uploading segments... Aug 27 21:02:25.127973293 (3h24m4.290574251s) [upload=3529]: Upload finished. err: metaclient: accounting backend: Redis pipeline exec failed: dial tcp: lookup redis-master.redis: operation was canceled Duration: 55.069118981s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..................X.....X.....XX....X......X...X...X........] duration: 26.096742639s FAIL (2025-08-27 21:02:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 26.096757167s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000601904s FAIL (2025-08-27 21:02:06 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000628383s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.161556171s FAIL (2025-08-27 21:01:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.161588723s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........................X....................] duration: 2.199921364s FAIL (2025-08-27 21:01:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.199949218s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...................X.....X.....XX....X......X...X...X.......] duration: 6.117449858s FAIL (2025-08-27 21:01:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.117475066s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........................X...................] duration: 1.954188771s FAIL (2025-08-27 21:01:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.954233908s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 6.077007947s FAIL (2025-08-27 21:00:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.077026602s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [X...................X.....X.....XX....X......X...X...X......] duration: 6.041072807s FAIL (2025-08-27 21:00:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.041099588s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.115165636s FAIL (2025-08-27 21:00:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.115202728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........................X..................] duration: 1.910749664s FAIL (2025-08-27 21:00:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 1.91078942s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [.X...................X.....X.....XX....X......X...X...X.....] duration: 6.251717975s FAIL (2025-08-27 20:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.251745429s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [................X.........................X.................] duration: 2.033238695s FAIL (2025-08-27 20:59:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.033268513s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [..X...................X.....X.....XX....X......X...X...X....] duration: 6.073643055s FAIL (2025-08-27 20:59:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.073669997s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.315671906s FAIL (2025-08-27 20:59:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.315712544s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 6.114691952s FAIL (2025-08-27 20:58:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer ======LOGS====== Aug 27 20:39:00.015972426 (3h0m39.178573364s) [upload=3126]: Starting upload Aug 27 20:39:00.016037420 (3h0m39.178638358s) [upload=3126]: Done starting upload Aug 27 20:39:00.016109899 (3h0m39.178710837s) [upload=3126]: Uploading segments... Aug 27 20:39:00.016467183 (3h0m39.179068121s) [upload=3126]: Got next segment. Inline: false Aug 27 20:39:00.016510967 (3h0m39.179111905s) [upload=3126]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:39:00.016515055 (3h0m39.179115993s) [upload=3126]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:39:15.495757690 (3h0m54.658358628s) [upload=3126]: Next segment err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Aug 27 20:39:15.495791945 (3h0m54.658392883s) [upload=3126]: Done uploading segments... Aug 27 20:39:15.495810181 (3h0m54.658411119s) [upload=3126]: Upload finished. err: metaclient: manager closed: closed: read tcp 188.166.225.170:44522->35.212.10.183:7777: read: connection reset by peer Duration: 6.114714274s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........................X................] duration: 2.060776224s FAIL (2025-08-27 20:58:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== Aug 27 20:49:07.986513533 (3h10m47.149114472s) [upload=3311]: Starting upload Aug 27 20:49:07.986560723 (3h10m47.149161661s) [upload=3311]: Done starting upload Aug 27 20:49:07.986608875 (3h10m47.149209813s) [upload=3311]: Uploading segments... Aug 27 20:49:07.986643141 (3h10m47.149244089s) [upload=3311]: Got next segment. Inline: true Aug 27 20:49:07.986646708 (3h10m47.149247647s) [upload=3311]: Waiting for error group managing segments... Aug 27 20:49:07.986680212 (3h10m47.14928115s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:49:07.986684230 (3h10m47.149285168s) [upload=3311]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 27 20:49:07.986686644 (3h10m47.149287572s) [upload=3311]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:49:08.210173316 (3h10m47.372774255s) [upload=3311]: Done uploading segments... Aug 27 20:49:08.210190419 (3h10m47.372791357s) [upload=3311]: Upload finished. err: metaclient: Too Many Requests Duration:* 2.060831419s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 9 out of 60 is failed (threshold=3) [...X...................X.....X.....XX....X......X...X...X...] duration: 6.062758719s FAIL (2025-08-27 20:58:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 51b22e4f55516154223240a6bde4ff3c8304909cbb99dd2bd21e7ca26cb10977 != before 137a5e8a70b6197715b8d247624b4d3341dda940ee028ced6e2ad52288433053 ======LOGS====== Aug 27 20:56:42.086600660 (3h18m21.249201598s) [upload=3442]: Starting upload Aug 27 20:56:42.086645605 (3h18m21.249246543s) [upload=3442]: Done starting upload Aug 27 20:56:42.086701482 (3h18m21.24930242s) [upload=3442]: Uploading segments... Aug 27 20:56:42.086840779 (3h18m21.249441717s) [upload=3442]: Got next segment. Inline: false Aug 27 20:56:42.086865055 (3h18m21.249465993s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 27 20:56:42.086869313 (3h18m21.249470241s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 27 20:56:42.669341974 (3h18m21.831942912s) [seg_pos={0 0}, upload=3442]: Begin upload segment... Aug 27 20:56:42.669381290 (3h18m21.831982228s) [seg_pos={0 0}, upload=3442]: Done begin upload segment. Aug 27 20:56:42.669402000 (3h18m21.832002938s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669415165 (3h18m21.832016093s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669490459 (3h18m21.832091467s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669760917 (3h18m21.832361855s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.669906455 (3h18m21.832507393s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670079375 (3h18m21.832680313s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670294127 (3h18m21.832895065s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670371966 (3h18m21.832972904s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670639167 (3h18m21.833240095s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670657411 (3h18m21.833258339s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670894996 (3h18m21.833495934s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.670913071 (3h18m21.833514009s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671136418 (3h18m21.833737356s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671240728 (3h18m21.833841656s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671438266 (3h18m21.834039194s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671697482 (3h18m21.83429842s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671700788 (3h18m21.834301726s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:42.671909728 (3h18m21.834510666s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Uploading piece... Aug 27 20:56:43.796911916 (3h18m22.959512854s) [upload=3442]: Next returned nil segment Aug 27 20:56:43.796927406 (3h18m22.959528514s) [upload=3442]: Waiting for error group managing segments... Aug 27 20:56:44.100353470 (3h18m23.262954408s) [noise=true , address=79.127.205.245:4249 , piece_id=2524OEXKVHHMUPCGIPB2CYLSDJKFAC7DHEYBJMOPYSHTCHHM5HJA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.106737816 (3h18m23.269338754s) [noise=true , address=79.127.205.234:4617 , piece_id=YJ3CUEXGGFTWO3SY7G2NVMR2VQQ7OLOY6XDHYHYLPLSAKCVKPOOQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.160182437 (3h18m23.322783365s) [noise=true , address=64.49.13.154:4935 , piece_id=CYKBDP4KCHHNOUL5GSU42TAJU5COF7N5LM6UGJL5KBWUS7FWUK2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.592691098 (3h18m23.755292026s) [noise=true , address=45.91.116.5:4324 , piece_id=BGA3MSLZBP43KVY3I4JAXTK5BFG6B53OZVGH2UWWHKUGNY2QMQIA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.601269813 (3h18m23.763870741s) [noise=true , address=45.91.116.8:4076 , piece_id=H4ND6MO5MPGDHEBTEWDHXFON462KHZ3NYPEY466PS7TA4DOFDRSA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.606287353 (3h18m23.768888282s) [noise=true , address=77.90.31.21:4886 , piece_id=POOGXVBEM2LT54CWBD2E7Q4AJEYHM4OVGVZLZU6TPZVNLURFSFMA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.610427395 (3h18m23.773028333s) [noise=true , address=45.91.116.11:4150 , piece_id=FTNZPG734LXGBYKAQGTR27GIKRWPLNDPMLBFTSTLOQQVW35A55JQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.741964718 (3h18m23.904565646s) [noise=true , address=104.36.82.23:4478 , piece_id=YVRCC26GSO42PN2EOIIWPF2P5W6W3SEHVCH5GG3OFLOWRSUMQMGQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.746854775 (3h18m23.909455713s) [noise=true , address=45.91.116.12:4242 , piece_id=QKEPIJC6EB5D7PPWDYDDQ6HUKWOOHMTMLD6F6DJKV7PTQNF7X6YA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.752911514 (3h18m23.915512442s) [noise=true , address=45.91.116.6:4984 , piece_id=7T5HFFSE4PN7JQSBWCZLP6W6WTZKQ4FHMFAVPIDIJXW2UJ5VYYEQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.761153954 (3h18m23.923754892s) [noise=true , address=77.90.31.12:4594 , piece_id=SIDWMABAUQKRUHZFXKE4P3N5BAMZPCIYDDHP5VBNAEWYNLEOYCRA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.765461947 (3h18m23.928062885s) [noise=true , address=77.90.31.15:4729 , piece_id=A2AL6OAKQECHEED2JTM3IUFYVV2ROKCY2FMXXDYYGBZIIMTSK2FQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.780738467 (3h18m23.943339395s) [noise=true , address=77.90.31.14:4680 , piece_id=RE5ONLQGMDJNGCSW7BCYYCGHBO5EYG2P6U535MBFC334IRBEIUAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781184401 (3h18m23.943785339s) [noise=true , address=77.90.31.10:4544 , piece_id=6KKAUPXK6FVS5UOQW6PNGI37MLGBXXLR5GI7QGC3FE544LPVAOEA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: <nil> Aug 27 20:56:44.781212214 (3h18m23.943813142s) [seg_pos={0 0}, upload=3442]: Segment reached optimal threshold of 14 pieces. Aug 27 20:56:44.781321974 (3h18m23.943922912s) [noise=true , address=104.36.82.36:4637 , piece_id=X5ITV3DJ62RHY3SAJLVMGIBCVZ4G537XNN4VKVYS6IZMS3REVRAQ, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G5MBgT9YVC7ug7m7a3gJtEvUZcxSLreNu91r6YT9TLZHJFsf6): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781353084 (3h18m23.943954022s) [noise=true , address=104.36.82.49:4992 , piece_id=WGOS4PDR347G66K63W4TX3IQFYM3PBHTL3T7PNTT3R2XDFQF7C7A, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hibaaBJsJFvJfrctxFTtx3uFzcM8JT5Rvy6UchdcnM1dJ28fv): protocol: expected piece hash; context canceled; EOF Aug 27 20:56:44.781458636 (3h18m23.944059574s) [noise=true , address=104.36.82.60:4366 , piece_id=XENBVOEPB366E6TLGN7FNGNELE5XOUSL7U43EZDYALTN7WTXHBWA, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NBUjBGU2MCjDx8iZKvzCvhL6GHCfdXfihC5H2uoBUmzhz9uoV): protocol: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37646->104.36.82.60:4366: use of closed network connection Aug 27 20:56:44.781473895 (3h18m23.944074823s) [noise=true , address=104.36.82.60:4367 , piece_id=E5PEX6DB62M4KC4QFG3QQR5F4PQX34RLVWDASCJUPTNVZXPVBW2Q, seg_pos={0 0}, upload=3442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34022->104.36.82.60:4367: use of closed network connection Aug 27 20:56:44.781497390 (3h18m23.944098318s) [seg_pos={0 0}, upload=3442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 27 20:56:44.781562023 (3h18m23.944162961s) [upload=3442]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 27 20:56:44.781566141 (3h18m23.944167079s) [upload=3442]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 27 20:56:45.104706146 (3h18m24.267307084s) [upload=3442]: Done uploading segments... Aug 27 20:56:45.104726776 (3h18m24.267327714s) [upload=3442]: Upload finished. err: <nil> Duration: 6.062800689s core uplink us-select-1 us1-placement-20 metainfo-projects

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)