Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................XXXX.XXXXX......................] duration:
5.974700412s
FAIL (2025-07-11 22:24:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.974726874s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................X..XX..........................] duration:
5.914234286s
FAIL (2025-07-11 22:24:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.914259514s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................XX.X..........................] duration:
5.52901838s
FAIL (2025-07-11 22:24:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.529039841s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................X..XX.........................] duration:
6.088852696s
FAIL (2025-07-11 22:24:19 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.088877853s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................X.XXX.XXXXX.......................] duration:
6.023324989s
FAIL (2025-07-11 22:24:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.02334675s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................X..XX........................] duration:
5.828666963s
FAIL (2025-07-11 22:23:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.828692171s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............................XXXX.XXXXX.....................] duration:
5.77904519s
FAIL (2025-07-11 22:23:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.779072022s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................X..XX.......................] duration:
6.908626423s
FAIL (2025-07-11 22:23:31 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.90866726s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................XX.X.........................] duration:
5.659603106s
FAIL (2025-07-11 22:23:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.659623174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................X.XXX.XXXXX......................] duration:
5.960163589s
FAIL (2025-07-11 22:23:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.96018474s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................X..XX......................] duration:
5.853014373s
FAIL (2025-07-11 22:22:51 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.853034702s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................XX.X........................] duration:
5.538619349s
FAIL (2025-07-11 22:22:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.5387087s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................X.XXX.XXXXX.....................] duration:
5.998131003s
FAIL (2025-07-11 22:22:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.998157514s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............................XXXX.XXXXX....................] duration:
5.814162803s
FAIL (2025-07-11 22:22:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.814178775s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................XX.X.......................] duration:
5.73422192s
FAIL (2025-07-11 22:21:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.734246857s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................X..XX.....................] duration:
5.881997803s
FAIL (2025-07-11 22:21:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.882027408s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............................XXXX.XXXXX...................] duration:
6.153456587s
FAIL (2025-07-11 22:21:46 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 6.153488257s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............................X.XXX.XXXXX....................] duration:
6.498865447s
FAIL (2025-07-11 22:21:30 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.498886316s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................XX.X......................] duration:
5.613595419s
FAIL (2025-07-11 22:21:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.613621469s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................XX.X.....................] duration:
5.713643041s
FAIL (2025-07-11 22:20:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.713665244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [................................XXXX.XXXXX..................] duration:
5.973102785s
FAIL (2025-07-11 22:20:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.973134296s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................X..XX....................] duration:
6.037691229s
FAIL (2025-07-11 22:20:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.037714081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............................X.XXX.XXXXX...................] duration:
6.162505897s
FAIL (2025-07-11 22:20:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.162528351s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............................X.XXX.XXXXX..................] duration:
6.48081482s
FAIL (2025-07-11 22:20:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.480837563s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................XX.X....................] duration:
5.56213543s
FAIL (2025-07-11 22:20:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.56215691s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................XXXX.XXXXX.................] duration:
5.845362144s
FAIL (2025-07-11 22:20:20 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.845389887s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................XX.X...................] duration:
5.538526299s
FAIL (2025-07-11 22:20:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.538563s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................X..XX...................] duration:
5.921299758s
FAIL (2025-07-11 22:19:56 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.921329816s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................................XXXX.XXXXX................] duration:
5.810305792s
FAIL (2025-07-11 22:19:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.810336729s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................................XXXX.XXXXX...............] duration:
5.928180128s
FAIL (2025-07-11 22:19:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.928207311s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [................................X.XXX.XXXXX.................] duration:
5.97004879s
FAIL (2025-07-11 22:19:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.970068408s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................................XX.X..................] duration:
5.667918723s
FAIL (2025-07-11 22:19:01 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.667940825s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X..XX..................] duration:
5.762102117s
FAIL (2025-07-11 22:18:59 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.762135081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [....................................XXXX.XXXXX..............] duration:
5.839233747s
FAIL (2025-07-11 22:18:44 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.839263274s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................................X..XX.................] duration:
6.059987563s
FAIL (2025-07-11 22:18:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.060013333s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................X.XXX.XXXXX................] duration:
6.313735755s
FAIL (2025-07-11 22:18:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.313792485s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................................X.XXX.XXXXX...............] duration:
6.452880623s
FAIL (2025-07-11 22:18:02 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.452925679s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................................XX.X.................] duration:
5.568335689s
FAIL (2025-07-11 22:17:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.568354875s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....................................XXXX.XXXXX.............] duration:
5.851135309s
FAIL (2025-07-11 22:17:42 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.851166962s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.......................................X..XX................] duration:
5.762733196s
FAIL (2025-07-11 22:17:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.762758213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...................................X.XXX.XXXXX..............] duration:
6.293163943s
FAIL (2025-07-11 22:17:22 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.293186334s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................................X..XX...............] duration:
6.178421364s
FAIL (2025-07-11 22:17:10 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.178447184s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [......................................XXXX.XXXXX............] duration:
5.964823597s
FAIL (2025-07-11 22:16:55 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.96485156s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [....................................X.XXX.XXXXX.............] duration:
6.261114873s
FAIL (2025-07-11 22:16:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.261138128s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [........................................XX.X................] duration:
5.576183887s
FAIL (2025-07-11 22:16:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.576414578s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................X..XX..............] duration:
5.648892946s
FAIL (2025-07-11 22:16:38 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.648920318s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.........................................XX.X...............] duration:
5.525959317s
FAIL (2025-07-11 22:16:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.525975187s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......................................XXXX.XXXXX...........] duration:
7.18283854s
FAIL (2025-07-11 22:16:09 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 7.182871051s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.....................................X.XXX.XXXXX............] duration:
6.126772202s
FAIL (2025-07-11 22:15:59 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.12680794s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................XX.X..............] duration:
6.432507622s
FAIL (2025-07-11 22:15:53 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.432530525s
core
uplink
us-select-1
us1-placement-20
|