Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [........................................XXXX.XXXXX..........] duration:
5.879399672s
FAIL (2025-07-11 22:15:36 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.879430262s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..........................................X..XX.............] duration:
5.889397793s
FAIL (2025-07-11 22:15:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.88943283s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................X..XX............] duration:
5.793426346s
FAIL (2025-07-11 22:15:07 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.793454228s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [......................................X.XXX.XXXXX...........] duration:
6.690344395s
FAIL (2025-07-11 22:15:07 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.690363572s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...........................................XX.X.............] duration:
5.610132058s
FAIL (2025-07-11 22:14:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.610148167s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................................X..XX...........] duration:
5.844313509s
FAIL (2025-07-11 22:14:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.844348044s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........................................XXXX.XXXXX.........] duration:
5.876330738s
FAIL (2025-07-11 22:14:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.876355926s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.......................................X.XXX.XXXXX..........] duration:
6.341414819s
FAIL (2025-07-11 22:14:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.341439906s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................................X..XX..........] duration:
5.8666171s
FAIL (2025-07-11 22:14:15 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.866645934s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [........................................X.XXX.XXXXX.........] duration:
6.389707776s
FAIL (2025-07-11 22:14:09 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.389730421s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............................................XX.X............] duration:
5.295167888s
FAIL (2025-07-11 22:13:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.295406865s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................................XXXX.XXXXX........] duration:
5.824198359s
FAIL (2025-07-11 22:13:32 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.824236893s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.........................................X.XXX.XXXXX........] duration:
6.294006342s
FAIL (2025-07-11 22:13:26 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.294032232s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................................X..XX.........] duration:
5.782568002s
FAIL (2025-07-11 22:13:13 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.782597709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............................................XX.X...........] duration:
5.554065544s
FAIL (2025-07-11 22:13:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.554087286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..........................................X.XXX.XXXXX.......] duration:
6.200709481s
FAIL (2025-07-11 22:13:08 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.200735921s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................................X.XXX.XXXXX......] duration:
6.071811239s
FAIL (2025-07-11 22:12:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.071833452s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................................X..XX........] duration:
6.395746702s
FAIL (2025-07-11 22:12:47 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.395774995s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............................................XX.X..........] duration:
5.512594056s
FAIL (2025-07-11 22:12:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.512619105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...........................................XXXX.XXXXX.......] duration:
6.449513412s
FAIL (2025-07-11 22:12:29 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 6.449537818s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...............................................XX.X.........] duration:
5.599172702s
FAIL (2025-07-11 22:12:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.599201917s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................................XXXX.XXXXX......] duration:
5.903724231s
FAIL (2025-07-11 22:11:53 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.903748688s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [............................................X.XXX.XXXXX.....] duration:
6.110795095s
FAIL (2025-07-11 22:11:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.110821825s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................................X..XX.......] duration:
6.295872002s
FAIL (2025-07-11 22:11:33 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 6.295898422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [................................................XX.X........] duration:
5.518375673s
FAIL (2025-07-11 22:11:24 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.518397575s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............................................XXXX.XXXXX.....] duration:
6.316518932s
FAIL (2025-07-11 22:10:50 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 6.316547806s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................................XX.X.......] duration:
5.655734363s
FAIL (2025-07-11 22:10:41 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.655750122s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.............................................X.XXX.XXXXX....] duration:
6.190212582s
FAIL (2025-07-11 22:10:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.190239273s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............................................XXXX.XXXXX....] duration:
10.685516141s
FAIL (2025-07-11 22:10:27 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 10.68554219s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.................................................X..XX......] duration:
7.490724405s
FAIL (2025-07-11 22:10:24 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 7.490745145s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..............................................X.XXX.XXXXX...] duration:
6.294423129s
FAIL (2025-07-11 22:10:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.294445734s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............................................XXXX.XXXXX...] duration:
6.937126973s
FAIL (2025-07-11 22:10:01 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 6.937154035s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [...............................................X.XXX.XXXXX..] duration:
6.643668726s
FAIL (2025-07-11 22:09:56 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.643685938s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................................XX.X......] duration:
7.47156333s
FAIL (2025-07-11 22:09:47 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.471582496s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..................................................X..XX.....] duration:
10.023387121s
FAIL (2025-07-11 22:09:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 10.023420546s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................................XX.X.....] duration:
7.146292908s
FAIL (2025-07-11 22:09:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.146312856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [................................................X.XXX.XXXXX.] duration:
6.460213222s
FAIL (2025-07-11 22:09:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.460242357s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [................................................XXXX.XXXXX..] duration:
5.718370767s
FAIL (2025-07-11 22:08:55 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 5.718398209s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [...................................................X..XX....] duration:
5.733313202s
FAIL (2025-07-11 22:08:45 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 5.733339733s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................................XXXX.XXXXX.] duration:
10.708841268s
FAIL (2025-07-11 22:08:40 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 10.708865113s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..................................................XXXX.XXXXX] duration:
30.233780984s
FAIL (2025-07-11 22:08:30 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 5)
======LOGS======
{0 0}, upload=355880]: Done begin upload segment.
Jul 11 22:08:00.702599502 (444h57m35.473240837s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702603690 (444h57m35.473245025s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.702956284 (444h57m35.473597619s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703119166 (444h57m35.473760501s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703227583 (444h57m35.473868918s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703520243 (444h57m35.474161578s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703530502 (444h57m35.474171837s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703797383 (444h57m35.474438718s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.703880252 (444h57m35.474521587s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704039647 (444h57m35.474680972s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.704271750 (444h57m35.474913075s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705204535 (444h57m35.47584587s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705557110 (444h57m35.476198445s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705766800 (444h57m35.476408126s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.705974929 (444h57m35.476616254s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:00.706188308 (444h57m35.476829643s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Uploading piece...
Jul 11 22:08:02.412760756 (444h57m37.183402091s) [upload=355880]: Next returned nil segment
Jul 11 22:08:02.412778691 (444h57m37.183420026s) [upload=355880]: Waiting for error group managing segments...
Jul 11 22:08:02.760975870 (444h57m37.531617205s) [noise=true , address=45.91.116.5:4299 , piece_id=FFMYICXNRW3JQEATBIYLA4RQYMHOJNS6K5JYYEQPZ2QPVQBHBK4A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.038510834 (444h57m37.809152169s) [noise=true , address=45.91.116.8:4048 , piece_id=PTOCSETB2T6GOH6PJFPRMA3MSQHFDJZUZ3UNACYWNZHE4ABHOTQQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.058278492 (444h57m37.828919817s) [noise=true , address=45.91.116.4:4939 , piece_id=FJGEG4J6LFZ2U4LNY5TXE3UDBIC6RDVBQXWFLIQIRREISYANC27Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.256226865 (444h57m38.0268682s) [noise=true , address=77.90.31.10:4517 , piece_id=63CMUBZO3H3WLGRJCALU4SEER3ZNUNEIGVSBFNNELEUFZTQ6ICEA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:03.258807431 (444h57m38.029448776s) [noise=true , address=77.90.31.14:4663 , piece_id=5UZDYRVAD376WG7G7SQNK6AUIWXZD2OXGXKBMQONP2DISM5U4RWA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: <nil>
Jul 11 22:08:29.815430122 (444h58m4.586071457s) [noise=true , address=104.36.82.8:4969 , piece_id=KKWP7BMHT6UK7I2JWZE3FBT6HMQ7M3TCXSKMNGO4O4VY4H46GUVA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815487462 (444h58m4.586128857s) [noise=true , address=104.36.82.13:4185 , piece_id=T3GZFCNWTIW7XP57KNZT7FZNGR3NCYFVDROVUCAPEDSHYDLQUL2A, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815517309 (444h58m4.586158644s) [noise=true , address=104.36.82.5:4837 , piece_id=7OB3NDK4P2WWZ7ZQLWCZBDF5GV7ONW4RUI6SIHHK5BPTIGE5ZGJQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815582673 (444h58m4.586224029s) [noise=true , address=104.36.82.6:4874 , piece_id=WLLMQZF6LUHKIX5HVGZQJS65GM3ZHITKEPZ3OFEZHPS4RGV5QTJA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815631728 (444h58m4.586273063s) [noise=true , address=104.36.82.7:4909 , piece_id=VWKDZYZK7KZA25FNRGBATEWICWUTMSQ2GI6ZQMDPRYOI6YOOI5VA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815657246 (444h58m4.586298581s) [noise=true , address=104.36.82.11:4114 , piece_id=DMTCSJKZKSDWYNY56FLDCC5D2IFDRUSLDKN4FSXX4ZC3OMKOYZTQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815691662 (444h58m4.586332997s) [noise=true , address=104.36.82.15:4258 , piece_id=3TBOXAOAPHNBVXJ3MBLWWO77JYTXSNVIITVLPHU76O5P6DB4NJ5Q, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: rpc: dial tcp 104.36.82.15:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815727029 (444h58m4.586368364s) [noise=true , address=104.36.82.15:4259 , piece_id=Q2CEYBJJZ7I5GNCKFY4LOTSMTB7O3MDA7VGK2EC4GAVWGRKATDXA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815755385 (444h58m4.58639672s) [noise=true , address=104.36.82.14:4222 , piece_id=33ICRA7TVVBN4Q3N4F2SFPY5UOTNCZBWUG5C2DT74JDEIJ5ASTCA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 188.166.225.170:49458->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815777556 (444h58m4.586418891s) [noise=true , address=104.36.82.4:4804 , piece_id=FKPFDFT5UBPVGEQORO34UGGM3F7SIM3CMMSA3V3GRIBXU3CBC2YA, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815783548 (444h58m4.586424894s) [noise=true , address=104.36.82.14:4223 , piece_id=737YQVUKZCEE2PPAPKKGBRYP6ASHSHOST4XGXEYHOSUZ3MLKCQDQ, seg_pos={0 0}, upload=355880]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 188.166.225.170:45820->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:08:29.815817934 (444h58m4.586459269s) [seg_pos={0 0}, upload=355880]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil>
Jul 11 22:08:30.047483492 (444h58m4.818124817s) [upload=355880]: Done uploading segments...
Jul 11 22:08:30.047507088 (444h58m4.818148423s) [upload=355880]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5)
Duration: 30.233802054s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................................XX.X....] duration:
6.622279455s
FAIL (2025-07-11 22:08:25 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.622299212s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [....................................................X..XX...] duration:
8.316467443s
FAIL (2025-07-11 22:08:22 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 8.316491671s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [.................................................X.XXX.XXXXX] duration:
30.000636384s
FAIL (2025-07-11 22:08:12 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000646393s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [..................................................X.XXX.XXXX] duration:
30.001034797s
FAIL (2025-07-11 22:07:42 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001060837s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................X..XX..] duration:
20.834809038s
FAIL (2025-07-11 22:07:35 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 11)
======LOGS======
47]: Done begin upload segment.
Jul 11 22:05:25.250192172 (444h55m0.020833497s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250231017 (444h55m0.020872352s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250569715 (444h55m0.02121105s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250766191 (444h55m0.021407526s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.250847707 (444h55m0.021489042s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251107715 (444h55m0.02174905s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251178610 (444h55m0.021819945s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251489936 (444h55m0.022131341s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251521937 (444h55m0.022163272s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.251908417 (444h55m0.022549742s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.252223779 (444h55m0.022865115s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253148930 (444h55m0.023790265s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253471086 (444h55m0.024112411s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.253751101 (444h55m0.024392436s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254046447 (444h55m0.024687802s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254400644 (444h55m0.025041979s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254676783 (444h55m0.025318108s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:25.254923645 (444h55m0.02556498s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Uploading piece...
Jul 11 22:05:26.394518518 (444h55m1.165159853s) [upload=355847]: Next returned nil segment
Jul 11 22:05:26.394534878 (444h55m1.165176213s) [upload=355847]: Waiting for error group managing segments...
Jul 11 22:05:26.700772093 (444h55m1.471413428s) [noise=true , address=79.127.205.233:4656 , piece_id=4YNKONSSBYFTV4CCGI3JKAWSADBEOBKKE24L7EEO64CWUZSFJSKQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:29.109783784 (444h55m3.880425109s) [noise=true , address=79.127.220.101:4059 , piece_id=5Y2CK5SM54UODEFCGPDTAAAMF3JRC5CV7AGQYCZ6PW47EQD5VV2Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:33.759174840 (444h55m8.529816165s) [noise=true , address=77.90.31.12:4619 , piece_id=V2CFGF4J3CRRMQVBVUTYFOAU6NHNW7ZZNXHLW3B5WC6RTW6MSJZQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:34.089773700 (444h55m8.860415035s) [noise=true , address=77.90.31.13:4640 , piece_id=DUQI7XZUGJFKCW5D73FZGEDSLR3S5PKCPPH6V2XLWSR35BUQZ54A, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:35.743647957 (444h55m10.514289292s) [noise=true , address=45.91.116.11:4149 , piece_id=JJ735AOMNAZ5V2BWJGA2BPLQ375574VANFAFBGXAOVNODKGD4K5Q, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.189380742 (444h55m15.960022077s) [noise=true , address=77.90.31.11:4588 , piece_id=QKDMMYO5KGDLTFB74BSPG7UYP6VWEUQBS52WPLW2VKDQHXM4LNOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.645213384 (444h55m16.415854719s) [noise=true , address=77.90.31.21:4910 , piece_id=R43OAJYZUOW3ZNG2PWF7BMXRO37U44MFEAGPEAPQCZNUP6GUEBIQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:41.866575910 (444h55m16.637217245s) [noise=true , address=45.91.95.2:4021 , piece_id=37GWHZ27ALNE3XJWYMRZ3QDB6HYL6JL5KSSRMR3BQMBMDEMQ6NAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:44.740074006 (444h55m19.510715341s) [noise=true , address=77.90.31.15:4707 , piece_id=IULW4EB4E6R56FTK3ZT527MYU22CQFW7XMWUC7ULUGJPS5VSL2ZA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:46.188108286 (444h55m20.958749621s) [noise=true , address=45.91.95.4:4021 , piece_id=NMZAYDJGGRU27TY7QCKYDGGFWBWMWWKHIROPT57QVUSC4HKAUHMQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:05:55.408676041 (444h55m30.179317376s) [noise=true , address=45.91.95.3:4021 , piece_id=INUXGJNMLAZM6NI5PRUDW37MH42AMIC2V4EESX3NLMNZKZ33ZQWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: <nil>
Jul 11 22:06:24.142005152 (444h55m58.912646487s) [noise=true , address=45.91.95.5:4034 , piece_id=CCLUI4ELT3AMG62RKTLXEGM4LBCQFTQQJHKKAD7LMOFE4XVYTWUA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1gzELQ4cJ6iXuHuRJTUE1gSGj2Wz2VYoqLdZpEzey5RQRvGQv8, address:45.91.95.5:4034): protocol: expected piece hash; context deadline exceeded; EOF
Jul 11 22:06:24.142454010 (444h55m58.913095375s) [noise=true , address=104.36.82.3:4799 , piece_id=E3PJM326VZOUQQIXYYK2T3IDEDGTHYR6M2GNPZECR4YXJRSJGCTA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12pMB9xsz6ArzVRzSkh59X8Y7v3w18CzWoVzBRhPdKZTne241v9, address:104.36.82.3:4799): protocol: write tcp 188.166.225.170:46732->104.36.82.3:4799: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142530917 (444h55m58.913172282s) [noise=true , address=104.36.82.6:4901 , piece_id=SE4Y3VRZRUWDBBIQFYCRDH3KYSPWFSN4KOCS3KKUMLLTDRX7S3PA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1SRXGomdNGRrsPBzD8YU34Y1EiuiJdnwuB79v8cMSLgTd7jpei, address:104.36.82.6:4901): protocol: write tcp 188.166.225.170:41510->104.36.82.6:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142582606 (444h55m58.913223991s) [noise=true , address=104.36.82.19:4414 , piece_id=4YUT5MJANGYRZUSAVBJ4NKZH5V5XTTMQI4OZX5WBZSMNS7IRJDPA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:1R6GJNS7bXfsDLsp3CE9qcvTbcT44QmgvSFzVbzpukyVYsezRQ, address:104.36.82.19:4414): protocol: write tcp 188.166.225.170:59368->104.36.82.19:4414: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142659403 (444h55m58.913300739s) [noise=true , address=104.36.82.4:4820 , piece_id=7DBP4GL3YMF23V3U27ZS6STMJWS2FLY4HWDKHBQL67A2C3SKUDWA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12TnUNGbwHSf335XW7vNuq1ffwhFkgxomckBQmF189779tRU59k, address:104.36.82.4:4820): protocol: write tcp 188.166.225.170:57218->104.36.82.4:4820: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142696935 (444h55m58.91333832s) [noise=true , address=104.36.82.51:4081 , piece_id=YQKYMQJ676ODM3ZTSHRQANURNQB4WKQS7HW34AB4G35FVYYC5FAA, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:121Gj3trM89mKpTffMYkrV9PrwkPqQU2LauCiG1v3J91Mej9KWF, address:104.36.82.51:4081): protocol: write tcp 188.166.225.170:43764->104.36.82.51:4081: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142743154 (444h55m58.913384489s) [noise=true , address=104.36.82.50:4035 , piece_id=RPDFNBKOYV4HTINO32MZ5K7ESLNDTUPSAHXKLKJ2J7XWWV7RJGOQ, seg_pos={0 0}, upload=355847]: Done uploading piece. err: ecclient: upload failed (node:12nevdZspAt1ce9XA2Emh8QJDxcmEKpYemCMwWidVi8RU6T6vZD, address:104.36.82.50:4035): protocol: rpc: dial tcp 104.36.82.50:4035: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:06:24.142774984 (444h55m58.91341635s) [seg_pos={0 0}, upload=355847]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil>
Jul 11 22:06:24.374341223 (444h55m59.144982558s) [upload=355847]: Done uploading segments...
Jul 11 22:06:24.374371692 (444h55m59.145013097s) [upload=355847]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11)
Duration: 20.834832692s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
8 out of 60 is failed (threshold=3) [...................................................XXXX.XXXX] duration:
30.23210605s
FAIL (2025-07-11 22:07:34 UTC)
orbiter warning:
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 8)
======LOGS======
tarting upload
Jul 11 22:07:04.500584501 (444h56m39.271225846s) [upload=355868]: Done starting upload
Jul 11 22:07:04.500760858 (444h56m39.271402193s) [upload=355868]: Uploading segments...
Jul 11 22:07:04.500789072 (444h56m39.271430407s) [upload=355868]: Got next segment. Inline: false
Jul 11 22:07:04.500811586 (444h56m39.271452921s) [upload=355868]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jul 11 22:07:04.500816254 (444h56m39.271457579s) [upload=355868]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jul 11 22:07:04.735319972 (444h56m39.505961307s) [seg_pos={0 0}, upload=355868]: Begin upload segment...
Jul 11 22:07:04.735379345 (444h56m39.5060207s) [seg_pos={0 0}, upload=355868]: Done begin upload segment.
Jul 11 22:07:04.735415885 (444h56m39.50605724s) [noise=true , address=104.36.82.11:4113 , piece_id=PJ76UVQTN4YOH5WK6Z7AYZOOXA7LNITM3Y2GKXFNRO6IMDM4OPMQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.735471983 (444h56m39.506113328s) [noise=true , address=104.36.82.6:4875 , piece_id=3XFZRTUJSHULLMX65ATIU5Z2GEV26OSG6WCKOMNNECXIQJ6E2TWQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.736096467 (444h56m39.506737792s) [noise=true , address=77.90.31.10:4517 , piece_id=G6YSAI2EV4RBDBVFKQ7KUKRULMBE5VCVVTSAMRBXKUIW6IBP2BBA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.736197321 (444h56m39.506838656s) [noise=true , address=104.36.82.14:4222 , piece_id=BX3NL26QTFUYNNF4HFPZOTZIJO53QD54RYBVCLN2FZHLMR4DA2FQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.736641931 (444h56m39.507283266s) [noise=true , address=45.91.116.8:4047 , piece_id=SL52JTSO2TXFOCGCSFEA77BFUXHUYCUSGLT4XUDS6NF2DZ5L2RVA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.737042026 (444h56m39.507683371s) [noise=true , address=104.36.82.8:4970 , piece_id=YC4STMKVSIMZO2NXATUWNNL4HWTXKH75MOWETFKKNMBW5BKKQKBQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.737392717 (444h56m39.508034052s) [noise=true , address=77.90.31.13:4627 , piece_id=URBCLNAFVEE7OYZVL6KOMQUNDHKOQ67WK4QPVTILA2AHXQMXGP3A, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.737758116 (444h56m39.508399451s) [noise=true , address=77.90.31.11:4554 , piece_id=4RQMQ44PE6LD2YLXE5ZMAVY2P2QYYAL42A6QPTCKZLXPUQXGDJVA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.738242102 (444h56m39.508883437s) [noise=true , address=104.36.82.16:4295 , piece_id=BXIOPOVENMAHFXCZRGOMB4AF2TU2MWBHN7NWFY4GKNVQTMYED7AA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.738562305 (444h56m39.50920364s) [noise=true , address=104.36.82.15:4257 , piece_id=PZY4AFYUD3AUK6A2VF4A3EJGCLVH5ZYQF3WTZAMHP2MX7PIGACFQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.738869503 (444h56m39.509510838s) [noise=true , address=104.36.82.3:4767 , piece_id=WIWAEGX6P7VM3LNUJJ6RET72DPTQGQRTNCXU3FI33JAMM4HHIYUA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.739164677 (444h56m39.509806012s) [noise=true , address=45.91.116.13:4155 , piece_id=6E23AOEFD54YIMFCVLUN3QGSEOI44R5SHOMRTXG3IJ7IWBEWLYVQ, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.739461766 (444h56m39.510103101s) [noise=true , address=45.91.116.5:4299 , piece_id=HPQYGGNIHNEPPJGYTEIXTNNYVXB32X5S2XP4KVHZ5SZEHWWM22EA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.739705382 (444h56m39.510346717s) [noise=true , address=77.90.31.18:4805 , piece_id=WO3RZKKF5E2OT36VZQCOUOB3TB742OCJIEMMY4BQWNHWHVG4HZQA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.739942817 (444h56m39.510584142s) [noise=true , address=104.36.82.5:4840 , piece_id=AQHIEOCMC2KJD3VTKFDYCYEVTFATX27L3ZZBSGGX22OOLYCPTP6A, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:04.740193506 (444h56m39.510834841s) [noise=true , address=77.90.31.12:4592 , piece_id=USIXD7X43GMFI4IOJKQKRKXFWCNG3CZEIU4JWELRZ7ZC2YHJJ4QA, seg_pos={0 0}, upload=355868]: Uploading piece...
Jul 11 22:07:06.494904742 (444h56m41.265546077s) [upload=355868]: Next returned nil segment
Jul 11 22:07:06.494924751 (444h56m41.265566086s) [upload=355868]: Waiting for error group managing segments...
Jul 11 22:07:06.794216407 (444h56m41.564857742s) [noise=true , address=45.91.116.8:4047 , piece_id=SL52JTSO2TXFOCGCSFEA77BFUXHUYCUSGLT4XUDS6NF2DZ5L2RVA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:06.823773854 (444h56m41.594415179s) [noise=true , address=45.91.116.13:4155 , piece_id=6E23AOEFD54YIMFCVLUN3QGSEOI44R5SHOMRTXG3IJ7IWBEWLYVQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:07.220693587 (444h56m41.991334922s) [noise=true , address=45.91.116.5:4299 , piece_id=HPQYGGNIHNEPPJGYTEIXTNNYVXB32X5S2XP4KVHZ5SZEHWWM22EA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:09.058442054 (444h56m43.829083389s) [noise=true , address=77.90.31.13:4627 , piece_id=URBCLNAFVEE7OYZVL6KOMQUNDHKOQ67WK4QPVTILA2AHXQMXGP3A, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:10.334832075 (444h56m45.10547341s) [noise=true , address=77.90.31.10:4517 , piece_id=G6YSAI2EV4RBDBVFKQ7KUKRULMBE5VCVVTSAMRBXKUIW6IBP2BBA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:10.350870038 (444h56m45.121511363s) [noise=true , address=77.90.31.12:4592 , piece_id=USIXD7X43GMFI4IOJKQKRKXFWCNG3CZEIU4JWELRZ7ZC2YHJJ4QA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:12.016007617 (444h56m46.786648952s) [noise=true , address=77.90.31.11:4554 , piece_id=4RQMQ44PE6LD2YLXE5ZMAVY2P2QYYAL42A6QPTCKZLXPUQXGDJVA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:20.168980913 (444h56m54.939622278s) [noise=true , address=77.90.31.18:4805 , piece_id=WO3RZKKF5E2OT36VZQCOUOB3TB742OCJIEMMY4BQWNHWHVG4HZQA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: <nil>
Jul 11 22:07:33.844664383 (444h57m8.615305718s) [noise=true , address=104.36.82.6:4875 , piece_id=3XFZRTUJSHULLMX65ATIU5Z2GEV26OSG6WCKOMNNECXIQJ6E2TWQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844680132 (444h57m8.615321457s) [noise=true , address=104.36.82.3:4767 , piece_id=WIWAEGX6P7VM3LNUJJ6RET72DPTQGQRTNCXU3FI33JAMM4HHIYUA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 188.166.225.170:49576->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844725049 (444h57m8.615366374s) [noise=true , address=104.36.82.16:4295 , piece_id=BXIOPOVENMAHFXCZRGOMB4AF2TU2MWBHN7NWFY4GKNVQTMYED7AA, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844767080 (444h57m8.615408415s) [noise=true , address=104.36.82.14:4222 , piece_id=BX3NL26QTFUYNNF4HFPZOTZIJO53QD54RYBVCLN2FZHLMR4DA2FQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: rpc: dial tcp 104.36.82.14:4222: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844785504 (444h57m8.615426839s) [noise=true , address=104.36.82.5:4840 , piece_id=AQHIEOCMC2KJD3VTKFDYCYEVTFATX27L3ZZBSGGX22OOLYCPTP6A, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 188.166.225.170:33444->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844794522 (444h57m8.615435847s) [noise=true , address=104.36.82.11:4113 , piece_id=PJ76UVQTN4YOH5WK6Z7AYZOOXA7LNITM3Y2GKXFNRO6IMDM4OPMQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: rpc: dial tcp 104.36.82.11:4113: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844806174 (444h57m8.615447509s) [noise=true , address=104.36.82.8:4970 , piece_id=YC4STMKVSIMZO2NXATUWNNL4HWTXKH75MOWETFKKNMBW5BKKQKBQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: write tcp 188.166.225.170:43668->104.36.82.8:4970: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jul 11 22:07:33.844862181 (444h57m8.615503516s) [noise=true , address=104.36.82.15:4257 , piece_id=PZY4AFYUD3AUK6A2VF4A3EJGCLVH5ZYQF3WTZAMHP2MX7PIGACFQ, seg_pos={0 0}, upload=355868]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: context deadline exceeded; context deadline exceeded
Jul 11 22:07:33.844883672 (444h57m8.615525048s) [seg_pos={0 0}, upload=355868]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil>
Jul 11 22:07:34.074840011 (444h57m8.845481346s) [upload=355868]: Done uploading segments...
Jul 11 22:07:34.074867513 (444h57m8.845508848s) [upload=355868]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8)
Duration: 30.232129624s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.....................................................XX.X...] duration:
14.70068022s
FAIL (2025-07-11 22:07:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 14.700694677s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [......................................................XX.X..] duration:
38.223551596s
FAIL (2025-07-11 22:07:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 38.223574459s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
7 out of 60 is failed (threshold=3) [...................................................X.XXX.XXX] duration:
30.000518235s
FAIL (2025-07-11 22:07:11 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000539555s
edge
gateway
us-select-1
|