Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X.........X..........................XX.................] duration:
33.389410001s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:53:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 33.389425641s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [X............X................X.............................] duration:
7.476639041s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:53:33 UTC)
orbiter warning: context deadline exceeded
Duration: 7.476656393s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [XX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X.] duration:
5.856787055s trace:
https://jaeger.dev.storj.io/trace/41e69a21e7df56df
FAIL (2025-12-29 15:53:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA) context deadline exceeded
======LOGS======
ing segments...
Dec 29 15:52:33.113308645 (576h23m32.98439171s) [upload=505297]: Got next segment. Inline: false
Dec 29 15:52:33.113353531 (576h23m32.984436576s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:52:33.113359291 (576h23m32.984442346s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:52:33.356574745 (576h23m33.22765778s) [seg_pos={0 0}, upload=505297]: Begin upload segment...
Dec 29 15:52:33.356623657 (576h23m33.227706693s) [seg_pos={0 0}, upload=505297]: Done begin upload segment.
Dec 29 15:52:33.356670446 (576h23m33.227753491s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357312349 (576h23m33.228395384s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357783265 (576h23m33.2288663s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358259242 (576h23m33.229342277s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358696595 (576h23m33.22977964s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359130992 (576h23m33.230214027s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359562725 (576h23m33.23064576s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.361070435 (576h23m33.23215348s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.365131138 (576h23m33.236214193s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.367723854 (576h23m33.238806889s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.368542724 (576h23m33.239625759s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.371024630 (576h23m33.242107665s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.372466505 (576h23m33.24354954s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.373865559 (576h23m33.244948614s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.374480460 (576h23m33.245563495s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.375504771 (576h23m33.246587816s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:35.074628983 (576h23m34.945712028s) [upload=505297]: Next returned nil segment
Dec 29 15:52:35.074720957 (576h23m34.945804022s) [upload=505297]: Waiting for error group managing segments...
Dec 29 15:52:35.358007782 (576h23m35.229090817s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.358063257 (576h23m35.229146292s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.364004791 (576h23m35.235087836s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.365266804 (576h23m35.236349849s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.369359468 (576h23m35.240442503s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.372684949 (576h23m35.243767984s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.376723029 (576h23m35.247806064s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.378716946 (576h23m35.249799981s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.387105650 (576h23m35.258188705s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.536904311 (576h23m35.407987356s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.581947891 (576h23m35.453030926s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.620922546 (576h23m35.492005591s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772505715 (576h23m35.64358876s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772528959 (576h23m35.643611994s) [seg_pos={0 0}, upload=505297]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:52:35.772730042 (576h23m35.643813087s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection
Dec 29 15:52:35.772738618 (576h23m35.643821653s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772748797 (576h23m35.643831843s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection
Dec 29 15:52:35.772759357 (576h23m35.643842402s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772907189 (576h23m35.643990235s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 15:52:35.772914934 (576h23m35.643997979s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.773065280 (576h23m35.644148325s) [seg_pos={0 0}, upload=505297]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:52:35.773149411 (576h23m35.644232456s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:52:35.773155192 (576h23m35.644238227s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:52:36.067027903 (576h23m35.938110958s) [upload=505297]: Done uploading segments...
Dec 29 15:52:36.067053292 (576h23m35.938136347s) [upload=505297]: Upload finished. err: <nil>
Duration: 5.856802885s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......X] duration:
30.002957407s trace:
https://jaeger.dev.storj.io/trace/41e69a21e7df56df
FAIL (2025-12-29 15:53:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ) context deadline exceeded; piecestore: (Node ID: 155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE, Piece ID: 2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA) context deadline exceeded
======LOGS======
ing segments...
Dec 29 15:52:33.113308645 (576h23m32.98439171s) [upload=505297]: Got next segment. Inline: false
Dec 29 15:52:33.113353531 (576h23m32.984436576s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:52:33.113359291 (576h23m32.984442346s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:52:33.356574745 (576h23m33.22765778s) [seg_pos={0 0}, upload=505297]: Begin upload segment...
Dec 29 15:52:33.356623657 (576h23m33.227706693s) [seg_pos={0 0}, upload=505297]: Done begin upload segment.
Dec 29 15:52:33.356670446 (576h23m33.227753491s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357312349 (576h23m33.228395384s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.357783265 (576h23m33.2288663s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358259242 (576h23m33.229342277s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.358696595 (576h23m33.22977964s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359130992 (576h23m33.230214027s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.359562725 (576h23m33.23064576s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.361070435 (576h23m33.23215348s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.365131138 (576h23m33.236214193s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.367723854 (576h23m33.238806889s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.368542724 (576h23m33.239625759s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.371024630 (576h23m33.242107665s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.372466505 (576h23m33.24354954s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.373865559 (576h23m33.244948614s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.374480460 (576h23m33.245563495s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:33.375504771 (576h23m33.246587816s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Uploading piece...
Dec 29 15:52:35.074628983 (576h23m34.945712028s) [upload=505297]: Next returned nil segment
Dec 29 15:52:35.074720957 (576h23m34.945804022s) [upload=505297]: Waiting for error group managing segments...
Dec 29 15:52:35.358007782 (576h23m35.229090817s) [noise=true , address=145.79.227.51:4411 , piece_id=PODPAJ2IEUQA6TJRC7FHOWNKWNWMDPIPG3QXDI3IY7B2IWVECDPQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.358063257 (576h23m35.229146292s) [noise=true , address=145.79.227.45:4230 , piece_id=KIYGXG5H35W4O47M3HBSI4KJQ3SYSD6NNJIAYLKBTMQCVEUIBLZA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.364004791 (576h23m35.235087836s) [noise=true , address=145.79.227.19:4078 , piece_id=CCYB7LPX6DJHKMRFVSM2D3UZWZH72WM4SF7XLJGPIXXC3RDD5POA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.365266804 (576h23m35.236349849s) [noise=true , address=145.79.227.50:4375 , piece_id=CTVQDPWDEPEOOSWL57JJ6CNIEKZOOAUOIJFLMNL2VPNV2VQ3MN3Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.369359468 (576h23m35.240442503s) [noise=true , address=145.79.227.48:4301 , piece_id=LLJIWWMZDZDJCODY7YJVQY27YQL4A36UBYEKGV47KJUREZTWFSCA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.372684949 (576h23m35.243767984s) [noise=true , address=145.79.227.11:4731 , piece_id=ZKTIGTQ4OJG3ZOV5R242FYWN23GJNMJOJQUSTKW2COX3WH2JGUEQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.376723029 (576h23m35.247806064s) [noise=true , address=145.79.227.9:4365 , piece_id=DZP53KNQIZ63XIJ7T65SCLBCQ2QJANM5MAX7AN54NR2YT2GY4U6A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.378716946 (576h23m35.249799981s) [noise=true , address=145.79.227.12:4765 , piece_id=GLPY57LL2VW5A6XJGNF5MSC2YVLYEWFOO3ADOXQIB4THEI3J7EFQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.387105650 (576h23m35.258188705s) [noise=true , address=145.79.227.47:4266 , piece_id=C6RV4UZS3CDOPNTJSBZGD4WWM53V7T27EAJ4IPRSCQ6KI2QBXTSA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.536904311 (576h23m35.407987356s) [noise=true , address=77.90.31.14:4663 , piece_id=2RXJROPWBPYOJB4XNNIVDCDUYUOKQONS77FKWLWGL3QLMGJ4SCLA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.581947891 (576h23m35.453030926s) [noise=true , address=77.90.31.21:4877 , piece_id=UYNEJM5O46TAC6D4QQRISIWWQAPVX2LKWD5C6MR5IMCYCIHCTCLQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.620922546 (576h23m35.492005591s) [noise=true , address=45.91.116.10:4081 , piece_id=3PU7WITYZNSEFDSETDJYOSCRYDBPRTWO5FOQEQWCLYZGX3DBYL6Q, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772505715 (576h23m35.64358876s) [noise=true , address=45.91.116.8:4047 , piece_id=7O57X2KVXEJ6SHF7Y6NC6B24ADJ45ZF5AA35QI4WXRSC3GS2YH4A, seg_pos={0 0}, upload=505297]: Done uploading piece. err: <nil>
Dec 29 15:52:35.772528959 (576h23m35.643611994s) [seg_pos={0 0}, upload=505297]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:52:35.772730042 (576h23m35.643813087s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48726->45.91.116.11:4117: use of closed network connection
Dec 29 15:52:35.772738618 (576h23m35.643821653s) [noise=true , address=45.91.116.11:4117 , piece_id=KHXD5HV5DUM2SECDMGZWJAUOQHDSBY2EPI2UTEGNJ57GVMMEUQNA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772748797 (576h23m35.643831843s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53796->77.90.31.10:4520: use of closed network connection
Dec 29 15:52:35.772759357 (576h23m35.643842402s) [noise=true , address=77.90.31.10:4520 , piece_id=VNFNZZWIGXOSCAQXQ6BMXY3CFS6PGE35UPBZYKB56IRJ3RS2ZLUQ, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.772907189 (576h23m35.643990235s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 15:52:35.772914934 (576h23m35.643997979s) [noise=true , address=145.79.227.33:4869 , piece_id=ID74MO4L7NZYFHMUWWRBIHSR5WJYK2SKWMGHBTIGFBKRWBBNWFBA, seg_pos={0 0}, upload=505297]: Long tail cancellation due to optimal threshold
Dec 29 15:52:35.773065280 (576h23m35.644148325s) [seg_pos={0 0}, upload=505297]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:52:35.773149411 (576h23m35.644232456s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:52:35.773155192 (576h23m35.644238227s) [upload=505297]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:52:36.067027903 (576h23m35.938110958s) [upload=505297]: Done uploading segments...
Dec 29 15:52:36.067053292 (576h23m35.938136347s) [upload=505297]: Upload finished. err: <nil>
Duration: 30.00297963s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X.........X..........................XX................] duration:
49.097045371s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:53:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 49.097062794s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X............X................X............................] duration:
14.174878752s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:52:49 UTC)
orbiter warning: context deadline exceeded
Duration: 14.174958622s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
44 out of 60 is failed (threshold=6) [X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXXX] duration:
5.001369791s trace:
https://jaeger.dev.storj.io/trace/15d5e1ae41af7706
FAIL (2025-12-29 15:52:35 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001393266s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......X.........X..........................XX...............] duration:
42.941247585s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:52:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 42.941264085s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X............X................X...........................] duration:
21.321715805s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:52:11 UTC)
orbiter warning: context deadline exceeded
Duration: 21.321740993s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
27 out of 60 is failed (threshold=3) [.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.......] duration:
5.728354487s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:52:08 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 5.728388451s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX......] duration:
5.969558533s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:51:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 5.969584672s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXXX] duration:
5.001026072s trace:
https://jaeger.dev.storj.io/trace/7d80face98cc4390
FAIL (2025-12-29 15:51:31 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001046571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......X.........X..........................XX..............] duration:
45.94225198s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:51:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 45.942280183s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [X..X............X................X..........................] duration:
9.257684964s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:51:24 UTC)
orbiter warning: context deadline exceeded
Duration: 9.25769891s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
28 out of 60 is failed (threshold=3) [.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.....] duration:
5.679352794s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:51:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 5.679376248s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [X.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXXX] duration:
5.001808373s trace:
https://jaeger.dev.storj.io/trace/8c13fe61dda5bbf
FAIL (2025-12-29 15:51:04 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00184366s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.X..X............X................X.........................] duration:
16.437831109s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:50:55 UTC)
orbiter warning: context deadline exceeded
Duration: 16.4378472s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........X.........X..........................XX.............] duration:
46.123308675s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:50:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 46.123340096s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX....] duration:
6.013637566s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:50:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 6.013672684s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [..X..X............X................X........................] duration:
14.223731173s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:50:28 UTC)
orbiter warning: context deadline exceeded
Duration: 14.223762443s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXXX] duration:
5.001724924s trace:
https://jaeger.dev.storj.io/trace/5858dd39155747da
FAIL (2025-12-29 15:50:15 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001744372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXXX] duration:
5.001121935s trace:
https://jaeger.dev.storj.io/trace/3833d52d8685e7b
FAIL (2025-12-29 15:49:52 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001143075s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.........X.........X..........................XX............] duration:
36.883843217s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:49:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 36.883954981s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [...X..X............X................X.......................] duration:
6.995292963s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:49:39 UTC)
orbiter warning: context deadline exceeded
Duration: 6.995314765s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
43 out of 60 is failed (threshold=6) [XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXXX] duration:
5.001355723s trace:
https://jaeger.dev.storj.io/trace/2c26b05865fb7cae
FAIL (2025-12-29 15:49:28 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001378004s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [.X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX...] duration:
6.390846103s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:49:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 6.390905085s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [....X..X............X................X......................] duration:
18.114712715s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:48:38 UTC)
orbiter warning: context deadline exceeded
Duration: 18.114741481s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXXX] duration:
5.001545904s trace:
https://jaeger.dev.storj.io/trace/5e7f8fb9f0485555
FAIL (2025-12-29 15:48:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001569269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [..X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX..] duration:
6.156950548s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:48:18 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 6.156975757s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........X.........X..........................XX...........] duration:
16.13843511s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:48:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 16.138456621s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....X..X............X................X.....................] duration:
8.92912345s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:47:54 UTC)
orbiter warning: context deadline exceeded
Duration: 8.929151884s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [X..........X.........X..........................XX..........] duration:
49.492698416s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:47:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 49.492724325s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [X.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXXX] duration:
5.000994594s trace:
https://jaeger.dev.storj.io/trace/15c9f22a176b7ea3
FAIL (2025-12-29 15:47:38 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001017126s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX.] duration:
6.040832049s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:47:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 6.040847567s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
30 out of 60 is failed (threshold=3) [X...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....XX] duration:
30.004489704s trace:
https://jaeger.dev.storj.io/trace/62d695f772d2c923
FAIL (2025-12-29 15:47:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ) context deadline exceeded; piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: 2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q) context deadline exceeded
======LOGS======
9 15:46:52.012526483 (576h17m51.883609518s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:52.257878835 (576h17m52.12896188s) [seg_pos={0 0}, upload=505207]: Begin upload segment...
Dec 29 15:46:52.257924402 (576h17m52.129007447s) [seg_pos={0 0}, upload=505207]: Done begin upload segment.
Dec 29 15:46:52.257965340 (576h17m52.129048375s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258025345 (576h17m52.12910839s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.258881073 (576h17m52.129964108s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259342944 (576h17m52.130425979s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259468192 (576h17m52.130551227s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259679734 (576h17m52.130762769s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259774555 (576h17m52.13085759s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.259995025 (576h17m52.13107806s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260031754 (576h17m52.131114789s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260311647 (576h17m52.131394682s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260382061 (576h17m52.131465096s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260581261 (576h17m52.131664296s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.260761284 (576h17m52.131844319s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261078858 (576h17m52.132161903s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261271065 (576h17m52.1323541s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:52.261395743 (576h17m52.132478778s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Uploading piece...
Dec 29 15:46:53.950272597 (576h17m53.821355632s) [upload=505207]: Next returned nil segment
Dec 29 15:46:53.950330197 (576h17m53.821413242s) [upload=505207]: Waiting for error group managing segments...
Dec 29 15:46:54.231461669 (576h17m54.102544714s) [noise=true , address=145.79.227.50:4375 , piece_id=NWQOEQNZE4PKVI6LREIFXDGJHMT6IJTDYATCOW2J2CS6QCOOR4YA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.237998074 (576h17m54.109081109s) [noise=true , address=145.79.227.7:4293 , piece_id=KX3RGWHZG2HC7AWAELGC2ZDHJKBIFLBGLBOMTGNF5OHW7WK3UBYQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.238784332 (576h17m54.109867377s) [noise=true , address=145.79.227.18:4006 , piece_id=J4T4KARWJOWJGN6WVYHI6XDUISBU4EG7R5FSPDQSJ5XPUBGJJYTQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.242722721 (576h17m54.113805756s) [noise=true , address=145.79.227.4:4187 , piece_id=PCLZOHCYYXAVEOU7IZF3S5RBFSNRVFMN3HAINUH6M2GYQ6KMBKUA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.243262169 (576h17m54.114345214s) [noise=true , address=145.79.227.16:4911 , piece_id=XLJMJR37EX7S7LQ7H252SCCBP6CZTKLPXVTGZKAG2B7COPSXBRAA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.248147311 (576h17m54.119230356s) [noise=true , address=145.79.227.41:4085 , piece_id=UAG7D3RI35VO4HYL5OMDSM2WXRX4TSE7ZAV3PBJYETF6QLVPZTKA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.257215387 (576h17m54.128298432s) [noise=true , address=145.79.227.19:4077 , piece_id=BWTXIEBVQ22NKGQZKSEEULWSHWDPOSPLCKCNSUDRUD2KYKWGIJJA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.266498985 (576h17m54.13758203s) [noise=true , address=145.79.227.38:4977 , piece_id=JYGGTHPXGE3EQSEO75XS6OJPUWNEPIRHEADLXGD3SOYVV53LZHGA, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.268626005 (576h17m54.13970904s) [noise=true , address=145.79.227.42:4122 , piece_id=CUMTMHMCONA3DKLGRFRU26P3O6SJWAE6DEQMM4TOA5QQIYZYUPNQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.511002361 (576h17m54.382085396s) [noise=true , address=45.91.116.10:4083 , piece_id=2L3BDYWFJVV4YS6Q7FQCW3JQEXBZOD2UKWOYVX73Z4KY2YYF3E3Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.620846191 (576h17m54.491929246s) [noise=true , address=45.91.116.13:4156 , piece_id=ASH6R5ZJL7RMEZCQZ46AGQSATK77DLC6AN3BM5BXRDAFT2OIUL5Q, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.651538264 (576h17m54.5226213s) [noise=true , address=45.91.116.8:4048 , piece_id=HBP2ZNAJZDOCO3EZ5WQWRT6HQT7X63WXUZXUX53BLPWFPNFZNOJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686910439 (576h17m54.557993474s) [noise=true , address=45.91.116.12:4225 , piece_id=ZCECBOERDBTBE7HSMSHDZ5SUN5W7QTH2ZDOJ7NNAHOGFK7CB6ZJQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: <nil>
Dec 29 15:46:54.686934555 (576h17m54.55801759s) [seg_pos={0 0}, upload=505207]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:54.687181495 (576h17m54.558264571s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35150->77.90.31.19:4842: use of closed network connection
Dec 29 15:46:54.687191113 (576h17m54.558274149s) [noise=true , address=77.90.31.19:4842 , piece_id=Q42HER6SR3QHTHGZDKXY3H7FNWKRCNWEGZE4QVFNJCMH2YUQ7IPQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687493219 (576h17m54.558576284s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43440->45.91.116.5:4300: use of closed network connection
Dec 29 15:46:54.687502887 (576h17m54.558585942s) [noise=true , address=45.91.116.5:4300 , piece_id=S4WFI4LQZ6UZD53LTGO3YR6FANWR6T7HV7OF2LORJUVPRGNWM3RQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687626232 (576h17m54.558709278s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57586->77.90.31.15:4700: use of closed network connection
Dec 29 15:46:54.687633987 (576h17m54.558717032s) [noise=true , address=77.90.31.15:4700 , piece_id=JA5YT2BK676B4NVOCICHGSXNAAAE2VQZAPGW5HPXURUOKEYNE6MQ, seg_pos={0 0}, upload=505207]: Long tail cancellation due to optimal threshold
Dec 29 15:46:54.687653705 (576h17m54.55873674s) [seg_pos={0 0}, upload=505207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:54.687781197 (576h17m54.558864242s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:54.687788401 (576h17m54.558871436s) [upload=505207]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:55.021211986 (576h17m54.892295031s) [upload=505207]: Done uploading segments...
Dec 29 15:46:55.021237675 (576h17m54.89232072s) [upload=505207]: Upload finished. err: <nil>
Duration: 30.004598291s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [X.....X..X............X................X....................] duration:
17.459889738s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:47:04 UTC)
orbiter warning: context deadline exceeded
Duration: 17.459917732s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X..........X.........X..........................XX.........] duration:
35.939572765s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:46:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 35.93959681s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XX.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XXX] duration:
5.001845895s trace:
https://jaeger.dev.storj.io/trace/3f960267baac7c75
FAIL (2025-12-29 15:46:56 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00188547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [.X...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....X] duration:
30.003229434s trace:
https://jaeger.dev.storj.io/trace/14273bac1c2f0b8f
FAIL (2025-12-29 15:46:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: 55YECRYM224TBJRIB4SNWFDIKWBH7JJF7DRRWVOMDN2XNESAFZPA) context deadline exceeded; piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: XJA6O7P3NK7CIFJSV5UEBXA446OIRJHMLS3QJG5G3Y5ASW6UPOVA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: LND4WXZD7Y4B2VXO3MTGV3VXDISP6CCDNAYV64QEGXF726W7CZXA) context deadline exceeded
======LOGS======
Dec 29 15:46:03.046960398 (576h17m2.918043443s) [upload=505196]: Starting upload
Dec 29 15:46:03.047017556 (576h17m2.918100591s) [upload=505196]: Done starting upload
Dec 29 15:46:03.047084704 (576h17m2.918167749s) [upload=505196]: Uploading segments...
Dec 29 15:46:03.047368113 (576h17m2.918451158s) [upload=505196]: Got next segment. Inline: false
Dec 29 15:46:03.047417829 (576h17m2.918500874s) [upload=505196]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:46:03.047422648 (576h17m2.918505693s) [upload=505196]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:46:03.293745528 (576h17m3.164828603s) [seg_pos={0 0}, upload=505196]: Begin upload segment...
Dec 29 15:46:03.293928948 (576h17m3.165012013s) [seg_pos={0 0}, upload=505196]: Done begin upload segment.
Dec 29 15:46:03.293981097 (576h17m3.165064142s) [noise=true , address=45.91.116.4:4939 , piece_id=BUBJMA7FEFZMEJZYN3GDEWX4XDBCCAPKZYCFOQ4HUALGQ3IM3SDA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.294070227 (576h17m3.165153262s) [noise=true , address=45.91.116.9:4264 , piece_id=I3JN47YHUT4HAHCJKPIE3AXZMBQUNJ5YH7VSMXHLXNRVWZKPNOJQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.295398376 (576h17m3.166481731s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDXDJCENEV23RYJHK4XSP7O2ULE2TO74A4RGDXZ66J4G6CAA7ETA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.295979672 (576h17m3.167062727s) [noise=true , address=145.79.227.49:4338 , piece_id=3RUYA2EXR4VP55NVRRVLVEZBUKV4CKNQAVBCITEFGQY3VI7A5GCQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.296581347 (576h17m3.167664392s) [noise=true , address=145.79.227.2:4113 , piece_id=5Q6GMB2SGAZB6GV2ESJAQ3U352QJY2IGHTQFZSZZAR3KFTBM66WQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.297138188 (576h17m3.168221253s) [noise=true , address=145.79.227.45:4230 , piece_id=7IYP56SVLM3ZUAPK2Z26AQMQOJNYCRR4SADBOZGKAHMMHCRVNXIA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.298069341 (576h17m3.169152386s) [noise=true , address=77.90.31.14:4661 , piece_id=XJA6O7P3NK7CIFJSV5UEBXA446OIRJHMLS3QJG5G3Y5ASW6UPOVA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.299088692 (576h17m3.170171737s) [noise=true , address=77.90.31.10:4518 , piece_id=TTLMMETND32QCD245TKXXHUUZ3HGQQX4IVRA4AMSEKGJMIE3LYVQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.299552155 (576h17m3.1706352s) [noise=true , address=45.91.116.13:4155 , piece_id=7WBAUTER4ZIP6TJQX4WUTFHC2JODYJO7H4ZXH4DBJ4TNT3I3PWIQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.300145384 (576h17m3.17122844s) [noise=true , address=45.91.116.14:4192 , piece_id=LND4WXZD7Y4B2VXO3MTGV3VXDISP6CCDNAYV64QEGXF726W7CZXA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.300241527 (576h17m3.171324572s) [noise=true , address=145.79.227.17:4970 , piece_id=JDO4SAE34V72NI6CIIX5VX3EWFGTQZBA3KUVSSVG4HC4FKQ25V4Q, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.301351831 (576h17m3.172434876s) [noise=true , address=45.91.116.11:4120 , piece_id=55YECRYM224TBJRIB4SNWFDIKWBH7JJF7DRRWVOMDN2XNESAFZPA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.302249150 (576h17m3.173332195s) [noise=true , address=145.79.227.18:4006 , piece_id=F7BFFIOO37KZEVZGX6M4ENI2RQN3X6ZRFUUILHT33K3P45F22WUQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.304604365 (576h17m3.17568741s) [noise=true , address=145.79.227.33:4869 , piece_id=WMSFWH5NGXGVX2FP6CHM7DINON26PKTCHXMBY6BRQ53Y6IJ5PC4Q, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.306521015 (576h17m3.17760406s) [noise=true , address=145.79.227.14:4838 , piece_id=32RIEWEEKVQQWBXEQXD4B3AOIXBNJRWMXEA6JOP4LJVLBJMMJHYQ, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:03.308370957 (576h17m3.179454012s) [noise=true , address=145.79.227.38:4978 , piece_id=OR3Y3JP4FN7G5M3DECA6M2GJMR6HPMW4JIFLBTSN57IX4L5JOGHA, seg_pos={0 0}, upload=505196]: Uploading piece...
Dec 29 15:46:05.018191793 (576h17m4.889274839s) [upload=505196]: Next returned nil segment
Dec 29 15:46:05.018205921 (576h17m4.889288956s) [upload=505196]: Waiting for error group managing segments...
Dec 29 15:46:05.295404803 (576h17m5.166487849s) [noise=true , address=145.79.227.17:4970 , piece_id=JDO4SAE34V72NI6CIIX5VX3EWFGTQZBA3KUVSSVG4HC4FKQ25V4Q, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.296725639 (576h17m5.167808684s) [noise=true , address=145.79.227.14:4838 , piece_id=32RIEWEEKVQQWBXEQXD4B3AOIXBNJRWMXEA6JOP4LJVLBJMMJHYQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.298284477 (576h17m5.169367522s) [noise=true , address=145.79.227.38:4978 , piece_id=OR3Y3JP4FN7G5M3DECA6M2GJMR6HPMW4JIFLBTSN57IX4L5JOGHA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.300857477 (576h17m5.171940522s) [noise=true , address=145.79.227.18:4006 , piece_id=F7BFFIOO37KZEVZGX6M4ENI2RQN3X6ZRFUUILHT33K3P45F22WUQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.312623100 (576h17m5.183706146s) [noise=true , address=145.79.227.45:4230 , piece_id=7IYP56SVLM3ZUAPK2Z26AQMQOJNYCRR4SADBOZGKAHMMHCRVNXIA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.326048675 (576h17m5.19713172s) [noise=true , address=145.79.227.2:4113 , piece_id=5Q6GMB2SGAZB6GV2ESJAQ3U352QJY2IGHTQFZSZZAR3KFTBM66WQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.483035644 (576h17m5.354118689s) [noise=true , address=77.90.31.14:4661 , piece_id=XJA6O7P3NK7CIFJSV5UEBXA446OIRJHMLS3QJG5G3Y5ASW6UPOVA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.543595952 (576h17m5.414678987s) [noise=true , address=45.91.116.11:4120 , piece_id=55YECRYM224TBJRIB4SNWFDIKWBH7JJF7DRRWVOMDN2XNESAFZPA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:05.648295687 (576h17m5.519378722s) [noise=true , address=45.91.116.14:4192 , piece_id=LND4WXZD7Y4B2VXO3MTGV3VXDISP6CCDNAYV64QEGXF726W7CZXA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:06.016402434 (576h17m5.88748547s) [noise=true , address=45.91.116.12:4227 , piece_id=ZDXDJCENEV23RYJHK4XSP7O2ULE2TO74A4RGDXZ66J4G6CAA7ETA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:06.269843388 (576h17m6.140926433s) [noise=true , address=45.91.116.9:4264 , piece_id=I3JN47YHUT4HAHCJKPIE3AXZMBQUNJ5YH7VSMXHLXNRVWZKPNOJQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:06.317321605 (576h17m6.18840465s) [noise=true , address=77.90.31.10:4518 , piece_id=TTLMMETND32QCD245TKXXHUUZ3HGQQX4IVRA4AMSEKGJMIE3LYVQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:06.428176922 (576h17m6.299259967s) [noise=true , address=45.91.116.4:4939 , piece_id=BUBJMA7FEFZMEJZYN3GDEWX4XDBCCAPKZYCFOQ4HUALGQ3IM3SDA, seg_pos={0 0}, upload=505196]: Done uploading piece. err: <nil>
Dec 29 15:46:06.428198763 (576h17m6.299281838s) [seg_pos={0 0}, upload=505196]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:46:06.428660512 (576h17m6.299743557s) [noise=true , address=145.79.227.33:4869 , piece_id=WMSFWH5NGXGVX2FP6CHM7DINON26PKTCHXMBY6BRQ53Y6IJ5PC4Q, seg_pos={0 0}, upload=505196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 29 15:46:06.428671253 (576h17m6.299754298s) [noise=true , address=145.79.227.33:4869 , piece_id=WMSFWH5NGXGVX2FP6CHM7DINON26PKTCHXMBY6BRQ53Y6IJ5PC4Q, seg_pos={0 0}, upload=505196]: Long tail cancellation due to optimal threshold
Dec 29 15:46:06.428872335 (576h17m6.299955382s) [noise=true , address=145.79.227.49:4338 , piece_id=3RUYA2EXR4VP55NVRRVLVEZBUKV4CKNQAVBCITEFGQY3VI7A5GCQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN): protocol: expected piece hash; context canceled; EOF
Dec 29 15:46:06.428878969 (576h17m6.299962014s) [noise=true , address=145.79.227.49:4338 , piece_id=3RUYA2EXR4VP55NVRRVLVEZBUKV4CKNQAVBCITEFGQY3VI7A5GCQ, seg_pos={0 0}, upload=505196]: Long tail cancellation due to optimal threshold
Dec 29 15:46:06.429225118 (576h17m6.300308163s) [noise=true , address=45.91.116.13:4155 , piece_id=7WBAUTER4ZIP6TJQX4WUTFHC2JODYJO7H4ZXH4DBJ4TNT3I3PWIQ, seg_pos={0 0}, upload=505196]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF
Dec 29 15:46:06.429232702 (576h17m6.300315747s) [noise=true , address=45.91.116.13:4155 , piece_id=7WBAUTER4ZIP6TJQX4WUTFHC2JODYJO7H4ZXH4DBJ4TNT3I3PWIQ, seg_pos={0 0}, upload=505196]: Long tail cancellation due to optimal threshold
Dec 29 15:46:06.429259353 (576h17m6.300342388s) [seg_pos={0 0}, upload=505196]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:46:06.429332703 (576h17m6.300415748s) [upload=505196]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:46:06.429337652 (576h17m6.300420697s) [upload=505196]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:46:06.724755608 (576h17m6.595838653s) [upload=505196]: Done uploading segments...
Dec 29 15:46:06.724795303 (576h17m6.595878338s) [upload=505196]: Upload finished. err: <nil>
Duration: 30.003255455s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [.X.....X..X............X................X...................] duration:
26.567216512s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:46:29 UTC)
orbiter warning: context deadline exceeded
Duration: 26.5672469s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXX.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.XX] duration:
5.00156972s trace:
https://jaeger.dev.storj.io/trace/6c3c873f34c7cfbd
FAIL (2025-12-29 15:46:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001593384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X..........X.........X..........................XX........] duration:
50.85217797s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:45:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 50.852196615s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [..X.....X..X............X................X..................] duration:
8.968526749s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:45:45 UTC)
orbiter warning: context deadline exceeded
Duration: 8.968551527s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [X.X...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX....] duration:
5.853391218s trace:
https://jaeger.dev.storj.io/trace/1b8c4763d6d8ca27
FAIL (2025-12-29 15:45:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ) context deadline exceeded
======LOGS======
Dec 29 15:43:20.404654369 (576h14m20.275737404s) [upload=505153]: Starting upload
Dec 29 15:43:20.404713232 (576h14m20.275796277s) [upload=505153]: Done starting upload
Dec 29 15:43:20.404803464 (576h14m20.275886499s) [upload=505153]: Uploading segments...
Dec 29 15:43:20.404907793 (576h14m20.275990828s) [upload=505153]: Got next segment. Inline: false
Dec 29 15:43:20.405015267 (576h14m20.276098312s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:43:20.405021669 (576h14m20.276104704s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:43:20.661046785 (576h14m20.53212982s) [seg_pos={0 0}, upload=505153]: Begin upload segment...
Dec 29 15:43:20.661112129 (576h14m20.532195164s) [seg_pos={0 0}, upload=505153]: Done begin upload segment.
Dec 29 15:43:20.661153377 (576h14m20.532236412s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661179698 (576h14m20.532262733s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661641577 (576h14m20.532724612s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661973079 (576h14m20.533056124s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.662249525 (576h14m20.53333257s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663138738 (576h14m20.534221773s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663490638 (576h14m20.534573673s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663814866 (576h14m20.534897901s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664636239 (576h14m20.535719284s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664921552 (576h14m20.536004587s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666081151 (576h14m20.537164186s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666464651 (576h14m20.537547686s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.667516624 (576h14m20.538599659s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.668542557 (576h14m20.539625592s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.669666699 (576h14m20.540749734s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.671344674 (576h14m20.542427719s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:22.398274658 (576h14m22.269357693s) [upload=505153]: Next returned nil segment
Dec 29 15:43:22.398287522 (576h14m22.269370568s) [upload=505153]: Waiting for error group managing segments...
Dec 29 15:43:22.647395656 (576h14m22.518478701s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.650762999 (576h14m22.521846044s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.656126952 (576h14m22.527210007s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657077202 (576h14m22.528160247s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657875742 (576h14m22.528958777s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.658809711 (576h14m22.529892776s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.659914695 (576h14m22.53099774s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.665976367 (576h14m22.537059402s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.666429550 (576h14m22.537512595s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669081139 (576h14m22.540164174s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669114673 (576h14m22.540197718s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.831117213 (576h14m22.702200248s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954348736 (576h14m22.825431771s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954367001 (576h14m22.825450037s) [seg_pos={0 0}, upload=505153]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:43:22.954554839 (576h14m22.825637874s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.954561803 (576h14m22.825644838s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956096275 (576h14m22.82717931s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956107275 (576h14m22.82719031s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956269585 (576h14m22.82735262s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956277049 (576h14m22.827360094s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956298589 (576h14m22.827381624s) [seg_pos={0 0}, upload=505153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:43:22.956418708 (576h14m22.827501753s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:43:22.956423878 (576h14m22.827506913s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:43:23.255931762 (576h14m23.127014797s) [upload=505153]: Done uploading segments...
Dec 29 15:43:23.255953513 (576h14m23.127036558s) [upload=505153]: Upload finished. err: <nil>
Duration: 5.853421586s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [.X.X...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX...] duration:
5.692877635s trace:
https://jaeger.dev.storj.io/trace/1b8c4763d6d8ca27
FAIL (2025-12-29 15:45:14 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ) context deadline exceeded
======LOGS======
Dec 29 15:43:20.404654369 (576h14m20.275737404s) [upload=505153]: Starting upload
Dec 29 15:43:20.404713232 (576h14m20.275796277s) [upload=505153]: Done starting upload
Dec 29 15:43:20.404803464 (576h14m20.275886499s) [upload=505153]: Uploading segments...
Dec 29 15:43:20.404907793 (576h14m20.275990828s) [upload=505153]: Got next segment. Inline: false
Dec 29 15:43:20.405015267 (576h14m20.276098312s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:43:20.405021669 (576h14m20.276104704s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:43:20.661046785 (576h14m20.53212982s) [seg_pos={0 0}, upload=505153]: Begin upload segment...
Dec 29 15:43:20.661112129 (576h14m20.532195164s) [seg_pos={0 0}, upload=505153]: Done begin upload segment.
Dec 29 15:43:20.661153377 (576h14m20.532236412s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661179698 (576h14m20.532262733s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661641577 (576h14m20.532724612s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661973079 (576h14m20.533056124s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.662249525 (576h14m20.53333257s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663138738 (576h14m20.534221773s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663490638 (576h14m20.534573673s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663814866 (576h14m20.534897901s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664636239 (576h14m20.535719284s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664921552 (576h14m20.536004587s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666081151 (576h14m20.537164186s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666464651 (576h14m20.537547686s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.667516624 (576h14m20.538599659s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.668542557 (576h14m20.539625592s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.669666699 (576h14m20.540749734s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.671344674 (576h14m20.542427719s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:22.398274658 (576h14m22.269357693s) [upload=505153]: Next returned nil segment
Dec 29 15:43:22.398287522 (576h14m22.269370568s) [upload=505153]: Waiting for error group managing segments...
Dec 29 15:43:22.647395656 (576h14m22.518478701s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.650762999 (576h14m22.521846044s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.656126952 (576h14m22.527210007s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657077202 (576h14m22.528160247s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657875742 (576h14m22.528958777s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.658809711 (576h14m22.529892776s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.659914695 (576h14m22.53099774s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.665976367 (576h14m22.537059402s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.666429550 (576h14m22.537512595s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669081139 (576h14m22.540164174s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669114673 (576h14m22.540197718s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.831117213 (576h14m22.702200248s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954348736 (576h14m22.825431771s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954367001 (576h14m22.825450037s) [seg_pos={0 0}, upload=505153]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:43:22.954554839 (576h14m22.825637874s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.954561803 (576h14m22.825644838s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956096275 (576h14m22.82717931s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956107275 (576h14m22.82719031s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956269585 (576h14m22.82735262s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956277049 (576h14m22.827360094s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956298589 (576h14m22.827381624s) [seg_pos={0 0}, upload=505153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:43:22.956418708 (576h14m22.827501753s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:43:22.956423878 (576h14m22.827506913s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:43:23.255931762 (576h14m23.127014797s) [upload=505153]: Done uploading segments...
Dec 29 15:43:23.255953513 (576h14m23.127036558s) [upload=505153]: Upload finished. err: <nil>
Duration: 5.692905138s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXXX.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.X] duration:
5.000524587s trace:
https://jaeger.dev.storj.io/trace/3b14cade81fe0abe
FAIL (2025-12-29 15:45:14 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000550015s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X..........X.........X..........................XX.......] duration:
39.149820932s trace:
https://jaeger.dev.storj.io/trace/2ad7801a2a7b8b37
FAIL (2025-12-29 15:45:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
058826s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459355714 (576h9m15.330438759s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.459774402 (576h9m15.330857437s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460174624 (576h9m15.331257659s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.460554407 (576h9m15.331637463s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461067765 (576h9m15.33215081s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.461466504 (576h9m15.332549549s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.462392488 (576h9m15.333475523s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.463675851 (576h9m15.334759076s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.464879443 (576h9m15.335962488s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.466140233 (576h9m15.337223268s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.467152250 (576h9m15.338235285s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.468454140 (576h9m15.339537185s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470054137 (576h9m15.341137182s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:15.470571442 (576h9m15.341654487s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Uploading piece...
Dec 29 15:38:16.760369715 (576h9m16.63145276s) [upload=505080]: Next returned nil segment
Dec 29 15:38:16.760387739 (576h9m16.631470784s) [upload=505080]: Waiting for error group managing segments...
Dec 29 15:38:17.332310596 (576h9m17.203393631s) [noise=true , address=145.79.227.12:4775 , piece_id=DDRA5N3T4GCTAINEAY5JXI2SST3N5HOUAOETIFOVSZ3XNUNDGHSQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.552149336 (576h9m17.423232381s) [noise=true , address=77.90.31.13:4648 , piece_id=L575I6FDX32SDWJEWQMI64BUGMQJM7K4THIUYHTR4UWXOJ3SVIKQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.553662758 (576h9m17.424745803s) [noise=true , address=45.91.116.6:4987 , piece_id=5Q2J3WDTNAVQ3X7OQIG5DFOYK2JVB6MSZXD4WDBQJM2LM6FHJAKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.574974379 (576h9m17.446057424s) [noise=true , address=45.91.116.12:4251 , piece_id=3YJXYTREZKRHVD6TS7NI74SPYFM3BW64JZCSQ45RDHSWY4EUNYOQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.612645260 (576h9m17.483728295s) [noise=true , address=45.91.116.8:4060 , piece_id=64IEACVNBRDT7S5JGKEKRQX2MTCDDHR54JVSDBCOHZ3ZRGA56BMQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:17.980409245 (576h9m17.85149229s) [noise=true , address=193.149.164.143:4696 , piece_id=NCDYIYB4WOWAMFSF2KYRDIHWYHYXRGVJEWCZIBRCQVVJSN7IPBKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.287094835 (576h9m18.15817787s) [noise=true , address=145.79.227.30:4794 , piece_id=ODUABFNJVLO3RJIQKWQMISH23DDSWEIYDL26ZEIHKUPFCBY3QD5A, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.307993910 (576h9m18.179076955s) [noise=true , address=145.79.227.35:4512 , piece_id=EKC5QR53K546TATQ6JUDYT2TJ6JT54DEHYWDWYD35E6IIW2JVZLA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.309961547 (576h9m18.181044592s) [noise=true , address=145.79.227.44:4199 , piece_id=NP3ZT4TBUGORGDGLIE2DQLIEHU7CJQP7R5R4TI3CJ7FGALAXETEA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.324425479 (576h9m18.195508514s) [noise=true , address=145.79.227.40:4084 , piece_id=7E7OJRSF7R72AISCPICKKFLXOUOKYCN6SCYVWXBJBDRJE6JX7GGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.535163630 (576h9m18.406246675s) [noise=true , address=145.79.227.34:4911 , piece_id=7EYC5HCQNL7UNOBWXAQADBAS5IG4NDHPGAUMBRRMAKDOZU5PEZUQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.622800635 (576h9m18.49388367s) [noise=true , address=77.90.31.21:4883 , piece_id=DMG2J4BC4GKL5NCU7DR3Z44OD6COMI4WQQVEXIZC4CC52HQJR6ZQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.662884792 (576h9m18.533967837s) [noise=true , address=77.90.31.18:4836 , piece_id=QE5KQP7HS6UL7D4PPHUV75EBBCLYAKHR5VXPE7AHDMLGZBV6M3TQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690784067 (576h9m18.561867102s) [noise=true , address=45.91.116.10:4099 , piece_id=EZWORGEOT2TBCTADOLPATRIVYDQEA4NHU2HWLCOWKO3C2G4GS7EQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: <nil>
Dec 29 15:38:18.690803454 (576h9m18.561886489s) [seg_pos={0 0}, upload=505080]: Segment reached optimal threshold of 14 pieces.
Dec 29 15:38:18.691024445 (576h9m18.56210748s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1SdEfJoiiNeVAJ6QkDdqZRuuqgsWvB7LAvNc69HUapL8r8qscR): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691031709 (576h9m18.562114744s) [noise=true , address=77.90.31.16:4762 , piece_id=XDLI43E2ZYVLQGR4IRSUHC7KTLGLXNXPRJGRXV6U4L2PYF74CUGA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691174511 (576h9m18.562257556s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52994->64.49.13.169:4705: use of closed network connection
Dec 29 15:38:18.691181624 (576h9m18.562264659s) [noise=true , address=64.49.13.169:4705 , piece_id=YYIHYNPU4IEZPZL3ZPCYUOTQSDBYLN6M6DHQB5NRROOMGRW6MMDQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691278569 (576h9m18.562361604s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sGc7qmx3smXoPkf6zMMt57V8VhF1kHaPhExKCe873fEm9yUih): protocol: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:38290->145.79.227.3:4167: use of closed network connection
Dec 29 15:38:18.691297105 (576h9m18.56238015s) [noise=true , address=145.79.227.3:4167 , piece_id=OR2V2NQVADORJ6V3OQD5DTO744LO7OIKGMGN72SNRGTGQICGDTKA, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691439305 (576h9m18.56252234s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12i57FF97oEddYLZpNxSq5or9WLEquL8ZW1cVUuW4AThBbDqcCL): protocol: expected piece hash; context canceled; EOF
Dec 29 15:38:18.691447491 (576h9m18.562530526s) [noise=true , address=77.90.31.19:4857 , piece_id=MXBNHONPW5I27KGDXVBMHZFFO5HBLIW2FMIYBSITADOYTF5SFBBQ, seg_pos={0 0}, upload=505080]: Long tail cancellation due to optimal threshold
Dec 29 15:38:18.691468531 (576h9m18.562551566s) [seg_pos={0 0}, upload=505080]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 29 15:38:18.691533093 (576h9m18.562616128s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:38:18.691537873 (576h9m18.562620908s) [upload=505080]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:38:18.999331133 (576h9m18.870414168s) [upload=505080]: Done uploading segments...
Dec 29 15:38:18.999354868 (576h9m18.870437903s) [upload=505080]: Upload finished. err: <nil>
Duration: 39.149837733s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
5 out of 60 is failed (threshold=3) [...X.....X..X............X................X.................] duration:
21.306162761s trace:
https://jaeger.dev.storj.io/trace/37b601126de8b032
FAIL (2025-12-29 15:44:55 UTC)
orbiter warning: context deadline exceeded
Duration: 21.306187087s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
29 out of 60 is failed (threshold=3) [..X.X...X.X.XXX.XXX.XXX..X...XXX..XX.X..X..XX.XX...X.X..XX..] duration:
5.752082442s trace:
https://jaeger.dev.storj.io/trace/1b8c4763d6d8ca27
FAIL (2025-12-29 15:44:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: 5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ) context deadline exceeded; piecestore: (Node ID: 128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m, Piece ID: ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ) context deadline exceeded
======LOGS======
Dec 29 15:43:20.404654369 (576h14m20.275737404s) [upload=505153]: Starting upload
Dec 29 15:43:20.404713232 (576h14m20.275796277s) [upload=505153]: Done starting upload
Dec 29 15:43:20.404803464 (576h14m20.275886499s) [upload=505153]: Uploading segments...
Dec 29 15:43:20.404907793 (576h14m20.275990828s) [upload=505153]: Got next segment. Inline: false
Dec 29 15:43:20.405015267 (576h14m20.276098312s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 29 15:43:20.405021669 (576h14m20.276104704s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 29 15:43:20.661046785 (576h14m20.53212982s) [seg_pos={0 0}, upload=505153]: Begin upload segment...
Dec 29 15:43:20.661112129 (576h14m20.532195164s) [seg_pos={0 0}, upload=505153]: Done begin upload segment.
Dec 29 15:43:20.661153377 (576h14m20.532236412s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661179698 (576h14m20.532262733s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661641577 (576h14m20.532724612s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.661973079 (576h14m20.533056124s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.662249525 (576h14m20.53333257s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663138738 (576h14m20.534221773s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663490638 (576h14m20.534573673s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.663814866 (576h14m20.534897901s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664636239 (576h14m20.535719284s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.664921552 (576h14m20.536004587s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666081151 (576h14m20.537164186s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.666464651 (576h14m20.537547686s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.667516624 (576h14m20.538599659s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.668542557 (576h14m20.539625592s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.669666699 (576h14m20.540749734s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:20.671344674 (576h14m20.542427719s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Uploading piece...
Dec 29 15:43:22.398274658 (576h14m22.269357693s) [upload=505153]: Next returned nil segment
Dec 29 15:43:22.398287522 (576h14m22.269370568s) [upload=505153]: Waiting for error group managing segments...
Dec 29 15:43:22.647395656 (576h14m22.518478701s) [noise=true , address=145.79.227.16:4911 , piece_id=H57UCFTQHE67PPQ2WWT3QP4YTKTMRF5K62C2V5D2ID6WRRO6QZHQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.650762999 (576h14m22.521846044s) [noise=true , address=145.79.227.39:4015 , piece_id=AB4LQ3P46SXG42KFX6RM5ZCO7UPGRYB4PH3627EIL2XVYEDD55YQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.656126952 (576h14m22.527210007s) [noise=true , address=145.79.227.19:4079 , piece_id=NDWRMOFOOYDQYXFABSTCXZLLC2K2XDXT77R3NOAFDXNNWKR2I53Q, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657077202 (576h14m22.528160247s) [noise=true , address=145.79.227.15:4876 , piece_id=WHYQNRXVNB4E66R5AKRG52QJ5CSPP4NBOJBHQ4CS4Q4QHUIYJGSA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.657875742 (576h14m22.528958777s) [noise=true , address=145.79.227.50:4375 , piece_id=NPGKAPPCLJH4VUBUZMB4TLBVA5W6EGBR5K5XHTKQYHL4JYRCD7QA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.658809711 (576h14m22.529892776s) [noise=true , address=145.79.227.48:4302 , piece_id=AK4YCDV6NFLE4QBJBJKZNC2OYH7SLKURMMXAJ4GTACLEUYCU7EEA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.659914695 (576h14m22.53099774s) [noise=true , address=145.79.227.47:4267 , piece_id=S7S5KZ6Z6TY7VXTLKIUP6FSTBARKGQAIUSY4OITCBSHHYU7NBAWA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.665976367 (576h14m22.537059402s) [noise=true , address=145.79.227.51:4411 , piece_id=B5LMSNJP4XJURZSYHXZ543R2O5C7JLX2IIOKKN5B522DYERHWEMQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.666429550 (576h14m22.537512595s) [noise=true , address=145.79.227.6:4259 , piece_id=PF6QHADTRVFFZLFQDEPTFYQI4ZYUJZX3DFEQ66YKKGAHT2IPOVPA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669081139 (576h14m22.540164174s) [noise=true , address=145.79.227.52:4447 , piece_id=MZN2L7L4YJVIUW34K4BU2WASCDVDABAUUNA4CCMLAWY5THWZXZNQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.669114673 (576h14m22.540197718s) [noise=true , address=145.79.227.38:4979 , piece_id=HJFVTHHI63AAHL76ZQGMCIM4MK7SJP4HXVEPSXXQW66IL23IMJRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.831117213 (576h14m22.702200248s) [noise=true , address=77.90.31.10:4519 , piece_id=5X6OOLJ43N2V35WFCB6NUJT72CRJ24BXO3QN5YNJEVUQ5VX32CQQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954348736 (576h14m22.825431771s) [noise=true , address=45.91.116.9:4264 , piece_id=ECWGPVLNXTVKEBK6VBW3RU2EM27YPMBFHJXMZMP5DSH27SBQ7EUQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: <nil>
Dec 29 15:43:22.954367001 (576h14m22.825450037s) [seg_pos={0 0}, upload=505153]: Segment reached optimal threshold of 13 pieces.
Dec 29 15:43:22.954554839 (576h14m22.825637874s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.954561803 (576h14m22.825644838s) [noise=true , address=77.90.31.19:4844 , piece_id=RT6MHEIJMRYGWW2CXYFWAFI7HC7E5LX5SGIDHEJBT3E3PDB7VVRQ, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956096275 (576h14m22.82717931s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956107275 (576h14m22.82719031s) [noise=true , address=45.91.116.5:4298 , piece_id=YGX6NFHJXB7UTHKWABXVOBNDCVYQQZWBD5CU3W4W2LJG5GOQ3QTA, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956269585 (576h14m22.82735262s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 29 15:43:22.956277049 (576h14m22.827360094s) [noise=true , address=145.79.227.33:4871 , piece_id=ZV63YYMEQZCDWBDNZOFGBWID6E3GSAODN7532RQNAXUNLFXWEW6A, seg_pos={0 0}, upload=505153]: Long tail cancellation due to optimal threshold
Dec 29 15:43:22.956298589 (576h14m22.827381624s) [seg_pos={0 0}, upload=505153]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 29 15:43:22.956418708 (576h14m22.827501753s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 29 15:43:22.956423878 (576h14m22.827506913s) [upload=505153]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 29 15:43:23.255931762 (576h14m23.127014797s) [upload=505153]: Done uploading segments...
Dec 29 15:43:23.255953513 (576h14m23.127036558s) [upload=505153]: Upload finished. err: <nil>
Duration: 5.752109593s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
42 out of 60 is failed (threshold=6) [XXXXX.XXXX.X.XX.XXX..X.XX..XXX..XX.XXX..XX..XXX.XXXXXXXXXXX.] duration:
5.001430242s trace:
https://jaeger.dev.storj.io/trace/4fd758579726c9be
FAIL (2025-12-29 15:44:33 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001454991s
core
uplink
us-select-1
us1-placement-20
|